...

View Full Version : Local Storage DB



RickP
11-28-2012, 01:04 AM
I'm new to javascript but not programming. I recently learned about local storage and love the idea of it. I'm creating a small website that let's me track my golf scores and other stats and I'm doing it 100% on the client side and storing the results in the local storage.

I work in databases all day long so I love that style more than the key/value that local storage offers so I set out to make a small library that converts local storage into a more DB structure. Thought I'd share with others to see what their thoughts are and maybe it'll be useful for someone.

- The table name is a key in the local storage
- The data is stored as an array of stringified JSON per table (key)
- The data is compressed and uncompressed when written to and read from local storage
- The first entry in an item is the table definition

Below is the usage with comments:

<script language="javascript" type="text/javascript">
$(document).ready(OnDocumentReady);

function OnDocumentReady() {
// create a database object which allows many database operations
var db = new LocalStorageDB();

// this is just for testing to clear the local storage while we test
db.TruncateAll();

// deletes the data in the players table
db.TruncateTable("players");

// check if the table exists
if (!db.TableExists("players")) {

// create a new table
var tblPlayers = new LocalStorageTable("players");

// add columns to the new table
tblPlayers.AddColumn("name");
tblPlayers.AddColumn("email");

// add the table to the database
db.AddTable(tblPlayers);
}

// insert some records
// first parameter is the table name to insert into
// second parameter is an object that has the same fields as the table definition
db.Insert("players", { name: "Bob", email: "bob@gmail.com" });
db.Insert("players", { name: "Joe", email: "joe@gmail.com" });
db.Insert("players", { name: "Rick", email: "rick@gmail.com" });
db.Insert("players", { name: "Mike", email: "mike@gmail.com" });
db.Insert("players", { name: "Kevan", email: "kevan@gmail.com" });

// this will update the name to Ricky where the name is currently Rick
// first param is table name
// second param is object of the fields you want to update with the value you want to update to
// third param is the where clause. a function that when the condition returns true, will update those records
db.Update("players", { name: "Ricky" }, function (v) { return v.name == "Rick"; });

// deletes when returns true in function
// first parameter is table name
// second parameter is the where clause. a function in where if the condition returns true the record will be deleted
db.Delete("players", function (v) { return v.name == "Mike"; });

// query recrods
// first param is the table name
// second param is the where clause. a function that when the condition returns true, will return that record as part of an array
var rsNames = db.Query("players", function (v) { return v.name == "Rick" && v.email == "rick@gmail.com"; });

// the result is an array of objects that match the table structure. check it's length to determine if anything was returned. access it's
// records via an index, and it's columns via the field name defined when the table was created
if (rsNames.length > 0)
var email = rsNames[0].email;

// erase the table completely. records and structure
db.DropTable("players");
}

</script>



Library is attached.

Dormilich
11-28-2012, 08:08 AM
I would use an array for each row object. saves from using split().

something along

var table_name = [
{ field1: "data", field2: "data", field3: "data" },
{ field1: "data", field2: "data", field3: "data" },
{ field1: "data", field2: "data", field3: "data" },
{ field1: "data", field2: "data", field3: "data" }
];

niralsoni
11-28-2012, 01:19 PM
Looks great to work with. But just to keep in mind the browser memory (storage size) available to implement local JavaScript database. Below is the spec -



Browser Storage support Survives browser Survives browser Storage size
restart crash
------------- --------------- ---------------- ---------------- ------------
Chrome 4 Yes Yes Yes 5 MB
Firefox 3.6 Yes Yes Yes 5 MB
Firefox 3 Yes Yes Yes 5 MB
Firefox 2 Yes Yes Yes 5 MB
IE8 Yes Yes Yes 10 MB
IE7 Yes Yes Yes 128 kB
IE6 Yes Yes Yes 128 kB
Opera 10.50 Yes Yes No 5 MB
Safari 4 Yes Yes Yes 5 MB
Iphone Safari Yes Yes Yes 5 MB

RickP
11-28-2012, 01:29 PM
I would use an array for each row object. saves from using split().

something along

var table_name = [
{ field1: "data", field2: "data", field3: "data" },
{ field1: "data", field2: "data", field3: "data" },
{ field1: "data", field2: "data", field3: "data" },
{ field1: "data", field2: "data", field3: "data" }
];

But it has to be persistent. To do that I'm using local storage and local storage can only store key/value as strings not objects. So at some level this has to be stored as a string and split to pull the data out that we want. I do plan on doing what you are saying and storing it as a string so that JSON.parse() on the entire string will make an array of objects for me and save me from having to call split() and loop, but I have to imagine at some level JSON.parse() is doing some kind of splitting and looping somehow on it's own.

Dormilich
11-28-2012, 01:32 PM
but I have to imagine at some level JSON.parse() is doing some kind of splitting and looping somehow on it's own.

thats the whole point. why splitting yourself if JSON.parse() (resp. JSON.stringify()) already does the hard work for you?

RickP
11-28-2012, 01:35 PM
Looks great to work with. But just to keep in mind the browser memory (storage size) available to implement local JavaScript database. Below is the spec -



Browser Storage support Survives browser Survives browser Storage size
restart crash
------------- --------------- ---------------- ---------------- ------------
Chrome 4 Yes Yes Yes 5 MB
Firefox 3.6 Yes Yes Yes 5 MB
Firefox 3 Yes Yes Yes 5 MB
Firefox 2 Yes Yes Yes 5 MB
IE8 Yes Yes Yes 10 MB
IE7 Yes Yes Yes 128 kB
IE6 Yes Yes Yes 128 kB
Opera 10.50 Yes Yes No 5 MB
Safari 4 Yes Yes Yes 5 MB
Iphone Safari Yes Yes Yes 5 MB




Yeah I noticed this. My first intention of usage for this is for a golf score mobile website so the data is going to be very small. I want to use local storage because some golf courses are way out of the way and don't get great reception, so my golf mobile website is going to be 1 page that loads all it needs to, and never requires internet access outside of navigating to it first which a user could do when internet coverage is available and just keep the browser open for using it where coverage isn't available. I do then plan on having an option for them to upload the data to a server if they wish.

I think 5MB is a decent amount of memory when dealing with just strings. You can fit a ton of information into 5 MB of text.

RickP
11-28-2012, 01:41 PM
thats the whole point. why splitting yourself if JSON.parse() (resp. JSON.stringify()) already does the hard work for you?

Yeah, I agree. It does save my library from having to do heavy lifting. I'm new to javascript so I wasn't aware JSON.parse() was able to do the whole array parsing thing. I plan on storing it as comma separated JSON value string in localStorage now (instead of /r) and when I need to manipulate the data I'm going to prefix & append to a var string the brackets and pass that result to JSON.parse() to make my life easier.



var data = localStorage.getItem(tblName);
var objData = JSON.parse("[" + data + "]");


I think this would be easier than storing the brackets inside the string when having to insert new rows. I can just put a comma at the end and insert vs having to worry about that last bracket.

[EDIT]
Oh JSON.stringify() will also do this. Yeah I'll just use a temp array when inserting and pulling data out. Thanks for the tip!

RickP
11-28-2012, 02:46 PM
When I parse the local storage stored as array of JSON objects I don't get an array of objects, I get an array of JSON strings. Is that what I'm supposed to get or should I get an array of objects that the JSON represents?

I do notice now I have / in the JSON string where before I didn't.

When I JSON.parse the local storage for a table the result is:

[{"name":"name"},{"name":"Rick"}]

But that's just an array of JSON string values. Is that what it should do? That would mean I have to JSON.parse those too to get objects. I thought 1 JSON.parse() would do that automatically for me?

rnd me
11-28-2012, 03:12 PM
Browser Storage support Survives browser Survives browser Storage size
restart crash
------------- --------------- ---------------- ---------------- ------------

Iphone Safari Yes Yes Yes 5 MB



1. this is wrong. iphone, in an attempt to promote it's $$$ Apps over webapps, clears the contents of localStorage when the browser restarts or when the phone crashes or is turned off. don't read about it, test it.


2. OP: obviously, fellow coders want you to work JSON.stringify and JSON.parse into the routines for your DB so they don't have to call them each time.

3. you can use another domain to get another 5MB. This requires converting your code to an async version, but doing so provides obvious benefits. I can post a x-domain localStorage proxy if anyone's interested. it uses postMessage() and hashChange() to communicate with a hidden iframe.


4. if dealing with text, you can zip the code before saving it to localstorage. This helps a lot because, for example, Chrome's localStrorage is utf16, which means two bytes per char; a 50% waste for most JSON. By using zip, you can make better use of the full 16bits, not to mention the space savings from deflate. i can post an inline zipper if there's interest. the deflate is ~10kb and the inflate is ~6kb; obviously those weights can MORE than pay for themselves pretty quickly.

RickP
11-28-2012, 03:25 PM
1. This is scary. That defeats the entire purpose of local storage lol. Stupid apple :(

2. That's why I made a library like this. The users didn't have to parse anything themselves. The library does it for them. Just trying to clean the library up with this shorthand stuff, but it won't affect the user of the lib.

3. From what I remember reading using other domains to gain more storage was frowned on. I can see why. If I needed more space and was OK with going to the internet to get it (reaching out to other domains would require I assume hitting the internet), I'd just use ajax to send the data to be stored on the server.

4. Would love to have an example of this. Guessing though this would trade-off performance since when reading/writing to localStorage we'd have to zip/unzip the data each time. Would have to test to see how much overhead it adds.

Dormilich
11-28-2012, 03:31 PM
i can post an inline zipper if there's interest. the deflate is ~10kb and the inflate is ~6kb; obviously those weights can MORE than pay for themselves pretty quickly.
yepp, theres interest.

RickP
11-28-2012, 04:02 PM
Updated the library code on the first post.

- Changed to stringifying an array of objects to local storage so parsing can also be done 100% by JSON

- Added TruncateTable function

- Added Update function

rnd me
11-28-2012, 04:04 PM
3. From what I remember reading using other domains to gain more storage was frowned on. I can see why. If I needed more space and was OK with going to the internet to get it (reaching out to other domains would require I assume hitting the internet), I'd just use ajax to send the data to be stored on the server.

4. Would love to have an example of this. Guessing though this would trade-off performance since when reading/writing to localStorage we'd have to zip/unzip the data each time. Would have to test to see how much overhead it adds.

3. the spec frowns on subdomains specifically. if you use a working manifest in that HTML page's <HTML> tag, you will not need the internet to access the domains localStorage. As long as you have enough access to another domain to publish an HTML file, you can embed that html file from another domain and talk to it using x-domain methods like iframe.src="#"+JSON.stringify(data) and top.postMessage(data).



I am planning on making an abstracter that lets you list several domains, filling them as needed and keeping track of what's where to provide at least 50megs using the 10 or so domains i control.

if i can get volunteers to embed more html pages in other domains, that amount can grow by leaps and bounds. the hosting cost would be minimal: no server processing, and ~3kb of bandwidth for a new visitor, no repeat traffic.

Anyway, that's not done yet, but i'll throw what i have in the post a code section when it's ready. gimme a couple weeks; i'm swamped.


4. you got it, gimme a few mins to find it.

RickP
11-28-2012, 04:17 PM
3. I could add this to my database library that I have here also. It would 100% abstract where the data is stored and retrieved from. I'm confused as to how this works though (again new to web development. normally doing console/desktop stuff). What is a working manifest int he <HTML> tag look like and what does it do? Why do you need volunteers to embed html pages in other domains? Not really following how this all works, but I like the idea of it. I do get the impression though that if this caught on it would be a reason for browsers to stop supporting it as it's sort of exploiting storage and it sounds like on the web side here so much of how to store data on the client is up in the air. Standards seem to many and people seem to be arguing over what's the best and what to support. Your Apple example is another example of browsers starting to rebel against local storage.

4. Thanks. I would love to add this to my database library also to save even more space when storing the data.

rnd me
11-28-2012, 05:34 PM
Intro
By popular demand, here is how i "zip and unzip" strings in javascript. I did not write the orig, that was done a long time a go in a land far away. But, i have managed to squeeze 55kb into 8kb, and 20kb into 5kb, making the package useful in real-life. i made a peppering of other minor optimizations. My point is that even though i didn't come up with this, it's still as it stands below, my pride and joy. This was an ace up my sleeve that I decided to throw down on the table for all to see. X-mas comes early this year...





Code


Live Copy (http://danml.com/js/compression.js)



/* Copyright (C) 1999 Masanao Izumo <iz@onicos.co.jp>
* Version: 1.0.1
* LastModified: Dec 25 1999.
* updated and compressed by dandavis, 2012 (attribution only derivative)
* Presumed abandonware, but retain this notice for public use.
*/

// 5kb:
var inflate=function(){function U(){this[J]=this[R]=null}function V(){this.n=this.b=this.e=0,this.t=null}function K(h,v,u,b,a,p){this[D]=16,this.N_MAX=288,this[z]=0,this[C]=null,this.m=0;var q=B(this[D]+1),m,c,l,g,d,e,f,i=B(this[D]+1),j,k,A,w=new V,r=B(this[D]);g=B(this.N_MAX);var x,s=B(this[D]+1),n,t,y;y=this[C]=null;for(d=0;d<q[E];d++)q[d]=0;for(d=0;d<i[E];d++)i[d]=0;for(d=0;d<r[E];d++)r[d]=null;for(d=0;d<g[E];d++)g[d]=0;for(d=0;d<s[E];d++)s[d]=0;m=256<v?h[256]:this[D],j=h,k=0,d=v;do q[j[k]]++,k++;while(0<--d);if(q[0]==v)this[C]=null,this[z]=this.m=0;else{for(e=1;e<=this[D]&&0==q[e];e++);f=e,p<e&&(p=e);for(d=this[D];0!=d&&0==q[d];d--);l=d,p>d&&(p=d);for(n=1<<e;e<d;e++,n<<=1)if(0>(n-=q[e])){this[z]=2,this.m=p;return}if(0>(n-=q[d]))this[z]=2,this.m=p;else{q[d]+=n,s[1]=e=0,j=q,k=1;for(A=2;0<--d;)s[A++]=e+=j[k++];j=h,d=k=0;do 0!=(e=j[k++])&&(g[s[e]++]=d);while(++d<v);v=s[l],s[0]=d=0,j=g,k=0,g=-1,x=i[0]=0,A=null;for(t=0;f<=l;f++)for(h=q[f];0<h--;){for(;f>x+i[1+g];){x+=i[1+g],g++,t=(t=l-x)>p?p:t;if((c=1<<(e=f-x))>h+1){c-=h+1;for(A=f;++e<t&&!((c<<=1)<=q[++A]);)c-=q[A]}x+e>m&&x<m&&(e=m-x),t=1<<e,i[1+g]=e,A=B(t);for(c=0;c<t;c++)A[c]=new V;null==y?y=this[C]=new U:y=y[R]=new U,y[R]=null,y[J]=A,r[g]=A,0<g&&(s[g]=d,w.b=i[g],w.e=16+e,w.t=A,e=(d&(1<<x)-1)>>x-i[g],r[g-1][e].e=w.e,r[g-1][e].b=w.b,r[g-1][e].n=w.n,r[g-1][e].t=w.t)}w.b=f-x,k>=v?w.e=99:j[k]<u?(w.e=256>j[k]?16:15,w.n=j[k++]):(w.e=a[j[k]-u],w.n=b[j[k++]-u]),c=1<<f-x;for(e=d>>x;e<t;e+=c)A[e].e=w.e,A[e].b=w.b,A[e].n=w.n,A[e].t=w.t;for(e=1<<f-1;0!=(d&e);e>>=1)d^=e;for(d^=e;(d&(1<<x)-1)!=s[g];)x-=i[g],g--}this.m=i[1],this[z]=0!=n&&1!=l?1:0}}}function f(h){for(;y<h;)H|=(M[E]==S?-1:M.charCodeAt(S++)&255)<<y,y+=8}function i(h){return H&ba[h]}function l(h){H>>=h,y-=h}function N(h,v,u){var b,a,p;if(0==u)return 0;for(p=0;;){f(r),a=s[J][i(r)];for(b=a.e;16<b;){if(99==b)return-1;l(a.b),b-=16,f(b),a=a.t[i(b)],b=a.e}l(a.b);if(16==b)j&=F-1,h[v+p++]=n[j++]=a.n;else{if(15==b)break;f(b),k=a.n+i(b),l(b),f(I),a=T[J][i(I)];for(b=a.e;16<b;){if(99==b)return-1;l(a.b),b-=16,f(b),a=a.t[i(b)],b=a.e}l(a.b),f(b),G=j-a.n-i(b);for(l(b);0<k&&p<u;)k--,G&=F-1,j&=F-1,h[v+p++]=n[j++]=n[G++]}if(p==u)return u}return t=-1,p}function ca(h,j,k){var b,a,p,q,m,c,n,g=B(316);for(b=0;b<g[E];b++)g[b]=0;f(5),c=257+i(5),l(5),f(5),n=1+i(5),l(5),f(4),b=4+i(4),l(4);if(286<c||30<n)return-1;for(a=0;a<b;a++)f(3),g[W[a]]=i(3),l(3);for(;19>a;a++)g[W[a]]=0;r=7,a=new K(g,19,19,null,null,r);if(0!=a[z])return-1;s=a[C],r=a.m,q=c+n;for(b=p=0;b<q;)if(f(r),m=s[J][i(r)],a=m.b,l(a),a=m.n,16>a)g[b++]=p=a;else if(16==a){f(2),a=3+i(2),l(2);if(b+a>q)return-1;for(;0<a--;)g[b++]=p}else{17==a?(f(3),a=3+i(3),l(3)):(f(7),a=11+i(7),l(7));if(b+a>q)return-1;for(;0<a--;)g[b++]=0;p=0}r=da,a=new K(g,c,257,X,Y,r),0==r&&(a[z]=1);if(0!=a[z])return 1==a[z],-1;s=a[C],r=a.m;for(b=0;b<n;b++)g[b]=g[b+c];return I=ea,a=new K(g,n,0,Z,$,I),T=a[C],I=a.m,0==I&&257<c?-1:(1==a[z],0!=a[z]?-1:N(h,j,k))}function fa(h,v,u){var b,a;for(b=0;b<u&&(!L||-1!=t);){if(0<k){if(t!=ga)for(;0<k&&b<u;)k--,G&=F-1,j&=F-1,h[v+b++]=n[j++]=n[G++];else{for(;0<k&&b<u;)k--,j&=F-1,f(8),h[v+b++]=n[j++]=i(8),l(8);0==k&&(t=-1)}if(b==u)break}if(-1==t){if(L)break;f(1),0!=i(1)&&(L=!0),l(1),f(2),t=i(2),l(2),s=null,k=0}switch(t){case 0:a=h;var p=v+b,q=u-b,m=void 0,m=y&7;l(m),f(16),m=i(16),l(16),f(16);if(m!=(~H&65535))a=-1;else{l(16),k=m;for(m=0;0<k&&m<q;)k--,j&=F-1,f(8),a[p+m++]=n[j++]=i(8),l(8);a=(0==k&&(t=-1),m)}break;case 1:if(null!=s)a=N(h,v+b,u-b);else a:{a=h,p=v+b,q=u-b;if(null==O){for(var c=void 0,m=B(288),c=0;144>c;c++)m[c]=8;for(;256>c;c++)m[c]=9;for(;280>c;c++)m[c]=7;for(;288>c;c++)m[c]=8;P=7,c=new K(m,288,257,X,Y,P);if(0!=c[z]){a=(alert("HufBuild error: "+c[z]),-1);break a}O=c[C],P=c.m;for(c=0;30>c;c++)m[c]=5;Q=5,c=new K(m,30,0,Z,$,Q);if(1<c[z]){a=(O=null,alert("HufBuild error: "+c[z]),-1);break a}aa=c[C],Q=c.m}a=(s=O,T=aa,r=P,I=Q,N(a,p,q))}break;case 2:null!=s?a=N(h,v+b,u-b):a=ca(h,v+b,u-b);break;default:a=-1}if(-1==a)return L?0:-1;b+=a}return b}var B=Array,z="status",E="length",C="root",D="BMAX",J="list",R="next",Q,F=32768,ga=0,da=9,ea=6,n,j,O=null,aa,P,H,y,t,L,k,G,s,T,r,I,M,S,ba=[0,1,3,7,15,31,63,127,255,511,1023,2047,4095,8191,16383,32767,65535],X=[3,4,5,6,7,8,9,10,11,13,15,17,19,23,27,31,35,43,51,59,67,83,99,115,131,163,195,227,258,0,0],Y=[0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0,99,99],Z=[1,2,3,4,5,7,9,13,17,25,33,49,65,97,129,193,257,385,513,769,1025,1537,2049,3073,4097,6145,8193,12289, 16385,24577],$=[0,0,0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13],W=[16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15];return window.inflate=function(h){var f,i,b;null==n&&(n=B(2*F)),y=H=j=0,t=-1,L=!1,k=G=0,s=null,M=h,S=0,f=B(1024);for(h="";0<(i=fa(f,0,f[E]));)for(b=0;b<i;b++)h+=String.fromCharCode(f[b]);return M=null,h}}();


// 8kb:
var deflate=function(){var e,t,n,r,i=null,s,o,u,a,f,l,c,h,p,d,v,m,g,y,b,w,E,S,x,T,N,C,k,L,A,O,M,_,D,P,H,B,j,F,I,q,R,U,z,W,X,V,$ ,J,K,Q,G,Y,Z,et,tt,nt=function(){this.dl=this.fc=0},rt=function(){this.extra_bits=this.static_tree=t his.dyn_tree=null,this.max_code=this.max_length=this.elems=this.extra_base=0},it=function(e,t,n,r){t his.good_length=e,this.max_lazy=t,this.nice_length=n,this.max_chain=r},st=function(){this.next=null, this.len=0,this.ptr=[8192],this.off=0},ot=[0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0],ut=[0,0,0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13],at=[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,3,7],ft=[16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15],lt=[new it(0,0,0,0),new it(4,4,8,4),new it(4,5,16,8),new it(4,6,32,32),new it(4,4,16,16),new it(8,16,32,32),new it(8,16,128,128),new it(8,32,128,256),new it(32,128,258,1024),new it(32,258,258,4096)],ct=function(r){i[o+s++]=r;if(8192==o+s&&0!=s){var u,r=(null!=e?(u=e,e=e.next):u=new st,u.next=null,u.len=u.off=0,u);null==t?t=n=r:n=n.next=r,r.len=s-o;for(u=0;u<r.len;u++)r.ptr[u]=i[o+u];s=o=0}},ht=function(e){e&=65535,8190>o+s?(i[o+s++]=e&255,i[o+s++]=e>>>8):(ct(e&255),ct(e>>>8))},pt=function(){v=(v<<5^a[E+3-1]&255)&8191,m=c[32768+v],c[E&32767]=m,c[32768+v]=E},dt=function(e,t){Ot(t[e].fc,t[e].dl)},vt=function(e,t,n){return e[t].fc<e[n].fc||e[t].fc==e[n].fc&&R[t]<=R[n]},mt=function(e,t,n){var r;for(r=0;r<n&&tt<et.length;r++)e[t+r]=et.charCodeAt(tt++)&255;return r},gt=function(e){var t=N,n=E,r,i=w,s=32506<E?E-32506:0,o=E+258,u=a[n+i-1],f=a[n+i];w>=L&&(t>>=2);do if(r=e,a[r+i]==f&&a[r+i-1]==u&&a[r]==a[n]&&a[++r]==a[n+1]){n+=2,r++;do;while(a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&n<o);r=258-(o-n),n=o-258;if(r>i){S=e,i=r;if(258<=r)break;u=a[n+i-1],f=a[n+i]}}while((e=c[e&32767])>s&&0!=--t);return i},yt=function(){var e,t,n=65536-T-E;if(-1==n)n--;else if(65274<=E){for(e=0;32768>e;e++)a[e]=a[e+32768];S-=32768,E-=32768,d-=32768;for(e=0;8192>e;e++)t=c[32768+e],c[32768+e]=32768<=t?t-32768:0;for(e=0;32768>e;e++)t=c[e],c[e]=32768<=t?t-32768:0;n+=32768}x||(e=mt(a,E+T,n),0>=e?x=!0:T+=e)},bt=function(e,n,i){var f;if(!r){if(!x){p=h=0;var l,F;if(0==_[0].dl){P.dyn_tree=A,P.static_tree=M,P.extra_bits=ot,P.extra_base=257,P.elems=286,P.max_length=15,P.max _code=0,H.dyn_tree=O,H.static_tree=_,H.extra_bits=ut,H.extra_base=0,H.elems=30,H.max_length=15,H.max _code=0,B.dyn_tree=D,B.static_tree=null,B.extra_bits=at,B.extra_base=0,B.elems=19,B.max_length=7;for (F=l=B.max_code=0;28>F;F++){W[F]=l;for(f=0;f<1<<ot[F];f++)U[l++]=F}U[l-1]=F;for(F=l=0;16>F;F++){X[F]=l;for(f=0;f<1<<ut[F];f++)z[l++]=F}for(l>>=7;30>F;F++){X[F]=l<<7;for(f=0;f<1<<ut[F]-7;f++)z[256+l++]=F}for(f=0;15>=f;f++)j[f]=0;for(f=0;143>=f;)M[f++].dl=8,j[8]++;for(;255>=f;)M[f++].dl=9,j[9]++;for(;279>=f;)M[f++].dl=7,j[7]++;for(;287>=f;)M[f++].dl=8,j[8]++;xt(M,287);for(f=0;30>f;f++)_[f].dl=5,_[f].fc=Mt(f,5);Et()}for(f=0;8192>f;f++)c[32768+f]=0;C=lt[k].max_lazy,L=lt[k].good_length,N=lt[k].max_chain,d=E=0,T=mt(a,0,65536);if(0>=T)x=!0,T=0;else{for(x=!1;262>T&&!x;)yt();for(f=v=0;2>f;f++)v=(v<<5^a[f]&255)&8191}t=null,o=s=0,3>=k?(w=2,b=0):(b=2,y=0),u=!1}r=!0;if(0==T)return u=!0,0}if((f=wt(e,n,i))==i)e=i;else if(u)e=f;else{if(3>=k)for(;0!=T&&null==t;){pt(),0!=m&&32506>=E-m&&(b=gt(m),b>T&&(b=T));if(3<=b)if(F=Lt(E-S,b-3),T-=b,b<=C){b--;do E++,pt();while(0!=--b);E++}else E+=b,b=0,v=a[E]&255,v=(v<<5^a[E+1]&255)&8191;else F=Lt(0,a[E]&255),T--,E++;for(F&&(kt(0),d=E);262>T&&!x;)yt()}else for(;0!=T&&null==t;){pt(),w=b,g=S,b=2,0!=m&&w<C&&32506>=E-m&&(b=gt(m),b>T&&(b=T),3==b&&4096<E-S&&b--);if(3<=w&&b<=w){F=Lt(E-1-g,w-3),T-=w-1,w-=2;do E++,pt();while(0!=--w);y=0,b=2,E++,F&&(kt(0),d=E)}else 0!=y?(Lt(0,a[E-1]&255)&&(kt(0),d=E),E++,T--):(y=1,E++,T--);for(;262>T&&!x;)yt()}e=(0==T&&(0!=y&&Lt(0,a[E-1]&255),kt(1),u=!0),f+wt(e,f+n,i-f))}return e},wt=function(n,r,u){var a,f,l;for(a=0;null!=t&&a<u;){f=u-a,f>t.len&&(f=t.len);for(l=0;l<f;l++)n[r+a+l]=t.ptr[t.off+l];t.off+=f,t.len-=f,a+=f,0==t.len&&(f=t,t=t.next,f.next=e,e=f)}if(a==u)return a;if(o<s){f=u-a,f>s-o&&(f=s-o);for(l=0;l<f;l++)n[r+a+l]=i[o+l];o+=f,a+=f,s==o&&(s=o=0)}return a},Et=function(){var e;for(e=0;286>e;e++)A[e].fc=0;for(e=0;30>e;e++)O[e].fc=0;for(e=0;19>e;e++)D[e].fc=0;A[256].fc=1,Q=$=J=K=Y=Z=0,G=1},St=function(e,t){for(var n=F[t],r=t<<1;r<=I;){r<I&&vt(e,F[r+1],F[r])&&r++;if(vt(e,n,F[r]))break;F[t]=F[r],t=r,r<<=1}F[t]=n},xt=function(e,t){var n=[16],r=0,i;for(i=1;15>=i;i++)r=r+j[i-1]<<1,n[i]=r;for(r=0;r<=t;r++)i=e[r].dl,0!=i&&(e[r].fc=Mt(n[i]++,i))},Tt=function(e){var t=e.dyn_tree,n=e.static_tree,r=e.elems,i,s=-1,o=r;I=0,q=573;for(i=0;i<r;i++)0!=t[i].fc?(F[++I]=s=i,R[i]=0):t[i].dl=0;for(;2>I;)i=F[++I]=2>s?++s:0,t[i].fc=1,R[i]=0,Y--,null!=n&&(Z-=n[i].dl);e.max_code=s;for(i=I>>1;1<=i;i--)St(t,i);do i=F[1],F[1]=F[I--],St(t,1),n=F[1],F[--q]=i,F[--q]=n,t[o].fc=t[i].fc+t[n].fc,R[i]>R[n]+1?R[o]=R[i]:R[o]=R[n]+1,t[i].dl=t[n].dl=o,F[1]=o++,St(t,1);while(2<=I);F[--q]=F[1],o=e.dyn_tree,i=e.extra_bits;var r=e.extra_base,n=e.max_code,u=e.max_length,a=e.static_tree,f,l,c,h,p=0;for(l=0;15>=l;l++)j[l]=0;o[F[q]].dl=0;for(e=q+1;573>e;e++)f=F[e],l=o[o[f].dl].dl+1,l>u&&(l=u,p++),o[f].dl=l,f>n||(j[l]++,c=0,f>=r&&(c=i[f-r]),h=o[f].fc,Y+=h*(l+c),null!=a&&(Z+=h*(a[f].dl+c)));if(0!=p){do{for(l=u-1;0==j[l];)l--;j[l]--,j[l+1]+=2,j[u]--,p-=2}while(0<p);for(l=u;0!=l;l--)for(f=j[l];0!=f;)i=F[--e],i>n||(o[i].dl!=l&&(Y+=(l-o[i].dl)*o[i].fc,o[i].fc=l),f--)}xt(t,s)},Nt=function(e,t){var n,r=-1,i,s=e[0].dl,o=0,u=7,a=4;0==s&&(u=138,a=3),e[t+1].dl=65535;for(n=0;n<=t;n++)i=s,s=e[n+1].dl,++o<u&&i==s||(o<a?D[i].fc+=o:0!=i?(i!=r&&D[i].fc++,D[16].fc++):10>=o?D[17].fc++:D[18].fc++,o=0,r=i,0==s?(u=138,a=3):i==s?(u=6,a=3):(u=7,a=4))},Ct=function(e,t){var n,r=-1,i,s=e[0].dl,o=0,u=7,a=4;0==s&&(u=138,a=3);for(n=0;n<=t;n++)if(i=s,s=e[n+1].dl,!(++o<u&&i==s)){if(o<a){do dt(i,D);while(0!=--o)}else 0!=i?(i!=r&&(dt(i,D),o--),dt(16,D),Ot(o-3,2)):10>=o?(dt(17,D),Ot(o-3,3)):(dt(18,D),Ot(o-11,7));o=0,r=i,0==s?(u=138,a=3):i==s?(u=6,a=3):(u=7,a=4)}},kt=function(e){var t,n,r;r=E-d,V[K]=Q,Tt(P),Tt(H);var i;Nt(A,P.max_code),Nt(O,H.max_code),Tt(B);for(i=18;3<=i&&0==D[ft[i]].dl;i--);i=(Y+=3*(i+1)+14,i),t=Y+3+7>>3,n=Z+3+7>>3,n<=t&&(t=n);if(r+4<=t&&0<=d){Ot(0+e,3),_t(),ht(r),ht(~r);for(i=0;i<r;i++)ct(a[d+i])}else if(n==t)Ot(2+e,3),At(M,_);else{Ot(4+e,3),r=P.max_code+1,t=H.max_code+1,i+=1,Ot(r-257,5),Ot(t-1,5),Ot(i-4,4);for(n=0;n<i;n++)Ot(D[ft[n]].dl,3);Ct(A,r-1),Ct(O,t-1),At(A,O)}Et(),0!=e&&_t()},Lt=function(e,t){l[$++]=t,0==e?A[t].fc++:(e--,A[U[t]+256+1].fc++,O[(256>e?z[e]:z[256+(e>>7)])&255].fc++,f[J++]=e,Q|=G),G<<=1,0==($&7)&&(V[K++]=Q,Q=0,G=1);if(2<k&&0==($&4095)){var n=8*$,r=E-d,i;for(i=0;30>i;i++)n+=O[i].fc*(5+ut[i]);n>>=3;if(J<parseInt($/2)&&n<parseInt(r/2))return!0}return 8191==$||8192==J},At=function(e,t){var n,r,i=0,s=0,o=0,u=0,a,c;if(0!=$)do 0==(i&7)&&(u=V[o++]),r=l[i++]&255,0==(u&1)?dt(r,e):(a=U[r],dt(a+256+1,e),c=ot[a],0!=c&&(r-=W[a],Ot(r,c)),n=f[s++],a=(256>n?z[n]:z[256+(n>>7)])&255,dt(a,t),c=ut[a],0!=c&&(n-=X[a],Ot(n,c))),u>>=1;while(i<$);dt(256,e)},Ot=function(e,t){p>16-t?(h|=e<<p,ht(h),h=e>>16-p,p+=t-16):(h|=e<<p,p+=t)},Mt=function(e,t){var n=0;do n|=e&1,e>>=1,n<<=1;while(0<--t);return n>>1},_t=function(){8<p?ht(h):0<p&&ct(h),p=h=0};return window.deflate=function(s,o){var u,h;et=s,tt=0,"undefined"==typeof o&&(o=6),(u=o)?1>u?u=1:9<u&&(u=9):u=6,k=u,x=r=!1;if(null==i){e=t=n=null,i=[8192],a=[65536],f=[8192],l=[32832],c=Array(65536),A=[573];for(u=0;573>u;u++)A[u]=new nt;O=[61];for(u=0;61>u;u++)O[u]=new nt;M=[288];for(u=0;288>u;u++)M[u]=new nt;_=[30];for(u=0;30>u;u++)_[u]=new nt;D=[39];for(u=0;39>u;u++)D[u]=new nt;P=new rt,H=new rt,B=new rt,j=[16],F=[573],R=[573],U=Array(256),z=[512],W=[29],X=[30],V=Array(1024)}for(var p=[1024],d=[];0<(u=bt(p,0,p.length));){var v=[u];for(h=0;h<u;h++)v[h]=String.fromCharCode(p[h]);d[d.length]=v.join("")}return et=null,d.join("")}}()






Considerations
Note that some char codes cannot be zipped. I am not 100% sure why this is. ascii is 100% ok, but SOME intl char sometimes throw it. forgive me for not figuring this out in detail, feel free, but the issue is easy to work around using escape/unescape or window.btoa/atob where available/possible. I've never had an issue using it since i figured out that much about the problem.


if you are storing plain text and json, you're fine.
binary and intl chars will need dumbed-down before zipping.
Even in that case, even with the escape overhead, zipping still saves a lot of space; escape only fattens the zip ~10-20%.

here is a usage example, proof of operation, and a demo of the intl complication:


Usage


// run in firebug/devtools/"F12" on http://www.codingforums.com/showthread.php?t=283110
function testZip(str){
return inflate(deflate(str))==str;
}

testZip(document.head.outerHTML); // true
testZip(document.body.outerHTML); // false
testZip(escape(document.body.outerHTML)); // true



deflate() is akin to escape() and inflate() is akin to unescape().


Performance


compression:

BYTES string source // note
83,816 document.body.outerHTML.length // orig
18,941 deflate(document.body.outerHTML).length // (non-operational)
21,406 deflate(escape(document.body.outerHTML)).length //works


that's about 4:1 on fairly non-repetitive HTML. JSON usually give me 5:1, and HTML tables can go up to 20:1. Elegant plain text is lucky to get 3:1. You can expect at least a doubling of the amount you can store, so 10mb is the new 5mb.

CPU usage:
my timings on 30,000 bytes of pretty repetitive json, running on a cheap year-old desktop:

0.09ms deflate (30kb _> 1.3kb)
0.13ms inflate (1.3kb _> 30kb)




notes


even on a phone 1/100th the speed of my desktop, we are talking about 13ms to inflate the data. the iphone 4S tests at about 1/13th of my desktop for comparison. so, on the iphone, this should happen in ~2ms. jquery.js might take 10ms on that device, not a humanly-perceivable delay.


bigger payloads will take more time, in a linear fashion, but they will also compresses with more efficiency. 105kb when compressed is more than 5kb smaller than 100kb when compressed. in fact, it would likely add less than 5% to the compressed size because of existing repetition.


breaking the data into chunks can reduce CPU at the cost of disk.
it WILL take a noticeable while to deflate 5mb of text on a smartphone. If you are trying to do this often, or at a bad time, say, onkeypress, your app will grind to a halt. If you had 10 different 500kb packages, you would only have to touch ~1/10th the data, and your pauses would be 10th as long.



you can use deflate() solely in your build process and tuck inflate() into your distribute-able package for smaller downloads.


usually the additional inflate CPU overhead is less time than downloading the extra uncompressed data would have been. This remains true on "under-powered" smartphones because their network, even at 4g is slow.


keep in mind that shipping a zipped package will reduce the efficiency of server-based transparent gzip transfer encoding. This is for (hopefully) obvious reasons. it's not like it hurts, you're still saving a ton compared to the orig. the zip/zip overhead is only about 10%, and given the advantage of being able to persist the compressed version client side, i think it's almost always worth it. a lot of servers don't even gzip or cache JS files, so that is not an issue for many.


if you server doesn't support gzip at all, you can probably get better app loading perf by zipping all your CSS files into one string, and using a javascript "addCSSasString()" type method. same benefits for JS.



one cool thing is that is when you ship, say jQuery.js as a zipped string, you can save the whole lib to localStorage. with a conditional adder like yepnope or whatever, you can load jquery locally upon the next visit instead of pinging the server to check the HTTP cache status, or even downloading it each time on a non-caching server. i've measured many a big improvement on production sites where i've rolled the pattern out, despite a few naysayers slamming localStorage performance a couple years ago...

here is an example of deploying code like that: http://danml.com/js/?compression,mode=json
as opposed to http://danml.com/js/compression.js or http://danml.com/js/?compression

to use that url's specific functionality, you have to define App and App.X, but then, any and all externally-loaded code is available in App.X, which is one JSON.stringify() away from persistence.
you should modify you own code server to dish out the callbacks or containers that your projects use.



In conclusion, enjoy and let me know what you think of it.

RickP
11-28-2012, 07:05 PM
It worked.

RickP
11-28-2012, 07:51 PM
It fails when trying to add to localStorage the following deflated value:

"‹VLQ:J9‰•E@””œ_ZTœ
2jc"

I get "Invalid argument." Note there is a return character there so not sure if that's messing it up.

This came from the string:

"[{"id":"id","player":"player","course":"course"}]"


Any ideas why that would be? Nothing in that string seems odd. It does work for some other strings I've done.

rnd me
11-28-2012, 09:23 PM
It fails when trying to add to localStorage the following deflated value:

"‹VLQ:J9‰•E@””œ_ZTœ
2jc"

I get "Invalid argument." Note there is a return character there so not sure if that's messing it up.

This came from the string:

"[{"id":"id","player":"player","course":"course"}]"


Any ideas why that would be? Nothing in that string seems odd. It does work for some other strings I've done.

ok. i looked into it.

Firstly, "It" doesn't fail, this is an IE localStorage problem, not a string problem or an issue with my functions.

my functions are fine in IE 6+



it seems that IE, even IE10 has an issue with localStorage.
this is a bug with IE's localStoage, not the code i posted, but i'll try to help anyway.

in chrome and firefox, this is true:
localStorage['temp']=unescape("%01");
localStorage['temp']==unescape("%01")
but in IE, it's false (9+10) or throws (7+8)

so, we need to fix IE, and IE only, both the 8 and 9/10 branch. Fun!

this alternate routine works on 8+9+10:


var code=deflate('[{"id":"id","player":"player","course":"course"}]');

localStorage['_temp']=escape(code);
inflate(unescape(localStorage['_temp']));

you may want to use a custom save(key, value) function instead of just using localStorage[key]=value, so that you have a chance in your normal workflow to handle IE's specific needs. it's also a chance to bake-in a date and other machine-gathered meta about the stored values.

just so there's no confusion to anyone reading this a year from now, one last time; the functions work in IE, localStorage has minor compat issues that require special handling.

why IE can't localStorage all the chars possibilities i don't know, but i appreciate your pointing this out, it's good to know.

RickP
11-28-2012, 09:44 PM
Gotta love browser compatibility.

Would it hurt any to call escape() and unescape() for all browsers?


Thanks for finding this also.

rnd me
11-28-2012, 11:29 PM
Would it hurt any to call escape() and unescape() for all browsers?

won't hurt, its just a waste of space...

RickP
11-28-2012, 11:51 PM
I think I'll try it with escape() and unescape() for all browsers. Should still get some good compression savings, but it will make the code more manageable. Today it's IE that doesn't work, tomorrow it's something else. This compression is already way more savings than the plain data so it's a win/win I think.

The one thing I hate about web programming are these browser specific things. Drives me nuts (can you tell I'm new to this :) )

RickP
11-29-2012, 05:45 AM
Library updated on front page:

- Put library in .js inside zip file on first post

- Added rnd_me's compression code to save space

- Added Delete() function. Parameters are similar to the Query() function. First parameter is the table name, second parameter is a function. The function should return true on the records to be deleted.

- Added Delete() function example to first post



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum