Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Page 1 of 2 12 LastLast
Results 1 to 15 of 22
  1. #1
    New Coder
    Join Date
    Nov 2012
    Posts
    26
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Post Local Storage DB

    I'm new to javascript but not programming. I recently learned about local storage and love the idea of it. I'm creating a small website that let's me track my golf scores and other stats and I'm doing it 100% on the client side and storing the results in the local storage.

    I work in databases all day long so I love that style more than the key/value that local storage offers so I set out to make a small library that converts local storage into a more DB structure. Thought I'd share with others to see what their thoughts are and maybe it'll be useful for someone.

    - The table name is a key in the local storage
    - The data is stored as an array of stringified JSON per table (key)
    - The data is compressed and uncompressed when written to and read from local storage
    - The first entry in an item is the table definition

    Below is the usage with comments:
    Code:
    <script language="javascript" type="text/javascript">
        $(document).ready(OnDocumentReady);
    
        function OnDocumentReady() {
            // create a database object which allows many database operations
            var db = new LocalStorageDB();
    
            // this is just for testing to clear the local storage while we test
            db.TruncateAll();
    
            // deletes the data in the players table
            db.TruncateTable("players");
    
            // check if the table exists
            if (!db.TableExists("players")) {
    
                // create a new table
                var tblPlayers = new LocalStorageTable("players");
    
                // add columns to the new table
                tblPlayers.AddColumn("name");
                tblPlayers.AddColumn("email");
    
                // add the table to the database
                db.AddTable(tblPlayers);
            }
    
            // insert some records
            // first parameter is the table name to insert into
            // second parameter is an object that has the same fields as the table definition
            db.Insert("players", { name: "Bob", email: "bob@gmail.com" });
            db.Insert("players", { name: "Joe", email: "joe@gmail.com" });
            db.Insert("players", { name: "Rick", email: "rick@gmail.com" });
            db.Insert("players", { name: "Mike", email: "mike@gmail.com" });
            db.Insert("players", { name: "Kevan", email: "kevan@gmail.com" });
    
            // this will update the name to Ricky where the name is currently Rick
            // first param is table name
            // second param is object of the fields you want to update with the value you want to update to
            // third param is the where clause. a function that when the condition returns true, will update those records
            db.Update("players", { name: "Ricky" },  function (v) { return v.name == "Rick"; });
    
            // deletes when returns true in function
            // first parameter is table name
            // second parameter is the where clause. a function in where if the condition returns true the record will be deleted
            db.Delete("players", function (v) { return v.name == "Mike"; });
    
            // query recrods
            // first param is the table name
            // second param is the where clause. a function that when the condition returns true, will return that record as part of an array
            var rsNames = db.Query("players", function (v) { return v.name == "Rick" && v.email == "rick@gmail.com"; });
    
            // the result is an array of objects that match the table structure. check it's length to determine if anything was returned. access it's
            // records via an index, and it's columns via the field name defined when the table was created
            if (rsNames.length > 0)
                var email = rsNames[0].email;
    
            // erase the table completely. records and structure
            db.DropTable("players");
        }
    
    </script>


    Library is attached.
    Attached Files Attached Files
    Last edited by RickP; 11-29-2012 at 04:54 AM.

  • #2
    Senior Coder Dormilich's Avatar
    Join Date
    Jan 2010
    Location
    Behind the Wall
    Posts
    3,237
    Thanks
    12
    Thanked 340 Times in 336 Posts
    I would use an array for each row object. saves from using split().

    something along
    PHP Code:
    var table_name = [
        { 
    field1"data"field2"data"field3"data" },
        { 
    field1"data"field2"data"field3"data" },
        { 
    field1"data"field2"data"field3"data" },
        { 
    field1"data"field2"data"field3"data" }
    ]; 
    The computer is always right. The computer is always right. The computer is always right. Take it from someone who has programmed for over ten years: not once has the computational mechanism of the machine malfunctioned.
    André Behrens, NY Times Software Developer

  • #3
    Regular Coder
    Join Date
    Mar 2008
    Location
    London
    Posts
    152
    Thanks
    4
    Thanked 42 Times in 42 Posts
    Looks great to work with. But just to keep in mind the browser memory (storage size) available to implement local JavaScript database. Below is the spec -
    Code:
    
    Browser          Storage support    Survives browser    Survives browser    Storage size
                                        restart             crash
    -------------    ---------------    ----------------    ----------------    ------------
    Chrome 4         Yes                Yes                 Yes                 5 MB
    Firefox 3.6      Yes                Yes                 Yes                 5 MB
    Firefox 3        Yes                Yes                 Yes                 5 MB
    Firefox 2        Yes                Yes                 Yes                 5 MB
    IE8              Yes                Yes                 Yes                 10 MB
    IE7              Yes                Yes                 Yes                 128 kB
    IE6              Yes                Yes                 Yes                 128 kB
    Opera 10.50      Yes                Yes                 No                  5 MB
    Safari 4         Yes                Yes                 Yes                 5 MB
    Iphone Safari    Yes                Yes                 Yes                 5 MB
    

  • #4
    New Coder
    Join Date
    Nov 2012
    Posts
    26
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Dormilich View Post
    I would use an array for each row object. saves from using split().

    something along
    PHP Code:
    var table_name = [
        { 
    field1"data"field2"data"field3"data" },
        { 
    field1"data"field2"data"field3"data" },
        { 
    field1"data"field2"data"field3"data" },
        { 
    field1"data"field2"data"field3"data" }
    ]; 
    But it has to be persistent. To do that I'm using local storage and local storage can only store key/value as strings not objects. So at some level this has to be stored as a string and split to pull the data out that we want. I do plan on doing what you are saying and storing it as a string so that JSON.parse() on the entire string will make an array of objects for me and save me from having to call split() and loop, but I have to imagine at some level JSON.parse() is doing some kind of splitting and looping somehow on it's own.

  • #5
    Senior Coder Dormilich's Avatar
    Join Date
    Jan 2010
    Location
    Behind the Wall
    Posts
    3,237
    Thanks
    12
    Thanked 340 Times in 336 Posts
    Quote Originally Posted by RickP View Post
    but I have to imagine at some level JSON.parse() is doing some kind of splitting and looping somehow on it's own.
    that’s the whole point. why splitting yourself if JSON.parse() (resp. JSON.stringify()) already does the hard work for you?
    The computer is always right. The computer is always right. The computer is always right. Take it from someone who has programmed for over ten years: not once has the computational mechanism of the machine malfunctioned.
    André Behrens, NY Times Software Developer

  • #6
    New Coder
    Join Date
    Nov 2012
    Posts
    26
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by niralsoni View Post
    Looks great to work with. But just to keep in mind the browser memory (storage size) available to implement local JavaScript database. Below is the spec -
    Code:
    
    Browser          Storage support    Survives browser    Survives browser    Storage size
                                        restart             crash
    -------------    ---------------    ----------------    ----------------    ------------
    Chrome 4         Yes                Yes                 Yes                 5 MB
    Firefox 3.6      Yes                Yes                 Yes                 5 MB
    Firefox 3        Yes                Yes                 Yes                 5 MB
    Firefox 2        Yes                Yes                 Yes                 5 MB
    IE8              Yes                Yes                 Yes                 10 MB
    IE7              Yes                Yes                 Yes                 128 kB
    IE6              Yes                Yes                 Yes                 128 kB
    Opera 10.50      Yes                Yes                 No                  5 MB
    Safari 4         Yes                Yes                 Yes                 5 MB
    Iphone Safari    Yes                Yes                 Yes                 5 MB
    

    Yeah I noticed this. My first intention of usage for this is for a golf score mobile website so the data is going to be very small. I want to use local storage because some golf courses are way out of the way and don't get great reception, so my golf mobile website is going to be 1 page that loads all it needs to, and never requires internet access outside of navigating to it first which a user could do when internet coverage is available and just keep the browser open for using it where coverage isn't available. I do then plan on having an option for them to upload the data to a server if they wish.

    I think 5MB is a decent amount of memory when dealing with just strings. You can fit a ton of information into 5 MB of text.

  • #7
    New Coder
    Join Date
    Nov 2012
    Posts
    26
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Dormilich View Post
    that’s the whole point. why splitting yourself if JSON.parse() (resp. JSON.stringify()) already does the hard work for you?
    Yeah, I agree. It does save my library from having to do heavy lifting. I'm new to javascript so I wasn't aware JSON.parse() was able to do the whole array parsing thing. I plan on storing it as comma separated JSON value string in localStorage now (instead of /r) and when I need to manipulate the data I'm going to prefix & append to a var string the brackets and pass that result to JSON.parse() to make my life easier.


    Code:
    var data = localStorage.getItem(tblName);
    var objData = JSON.parse("[" + data + "]");

    I think this would be easier than storing the brackets inside the string when having to insert new rows. I can just put a comma at the end and insert vs having to worry about that last bracket.

    [EDIT]
    Oh JSON.stringify() will also do this. Yeah I'll just use a temp array when inserting and pulling data out. Thanks for the tip!
    Last edited by RickP; 11-28-2012 at 12:44 PM.

  • #8
    New Coder
    Join Date
    Nov 2012
    Posts
    26
    Thanks
    1
    Thanked 0 Times in 0 Posts
    When I parse the local storage stored as array of JSON objects I don't get an array of objects, I get an array of JSON strings. Is that what I'm supposed to get or should I get an array of objects that the JSON represents?

    I do notice now I have / in the JSON string where before I didn't.

    When I JSON.parse the local storage for a table the result is:

    [{"name":"name"},{"name":"Rick"}]

    But that's just an array of JSON string values. Is that what it should do? That would mean I have to JSON.parse those too to get objects. I thought 1 JSON.parse() would do that automatically for me?

  • #9
    Senior Coder rnd me's Avatar
    Join Date
    Jun 2007
    Location
    Urbana
    Posts
    4,292
    Thanks
    10
    Thanked 583 Times in 564 Posts
    Quote Originally Posted by niralsoni View Post
    Code:
    
    Browser          Storage support    Survives browser    Survives browser    Storage size
                                        restart             crash
    -------------    ---------------    ----------------    ----------------    ------------
    
    Iphone Safari    Yes                Yes                 Yes                 5 MB
    
    1. this is wrong. iphone, in an attempt to promote it's $$$ Apps over webapps, clears the contents of localStorage when the browser restarts or when the phone crashes or is turned off. don't read about it, test it.


    2. OP: obviously, fellow coders want you to work JSON.stringify and JSON.parse into the routines for your DB so they don't have to call them each time.

    3. you can use another domain to get another 5MB. This requires converting your code to an async version, but doing so provides obvious benefits. I can post a x-domain localStorage proxy if anyone's interested. it uses postMessage() and hashChange() to communicate with a hidden iframe.


    4. if dealing with text, you can zip the code before saving it to localstorage. This helps a lot because, for example, Chrome's localStrorage is utf16, which means two bytes per char; a 50% waste for most JSON. By using zip, you can make better use of the full 16bits, not to mention the space savings from deflate. i can post an inline zipper if there's interest. the deflate is ~10kb and the inflate is ~6kb; obviously those weights can MORE than pay for themselves pretty quickly.
    my site (updated 13/9/26)
    BROWSER STATS [% share] (2014/5/28) IE7:0.1, IE8:5.3, IE11:8.4, IE9:3.2, IE10:3.2, FF:18.2, CH:46, SF:7.9, NON-MOUSE:32%

  • #10
    New Coder
    Join Date
    Nov 2012
    Posts
    26
    Thanks
    1
    Thanked 0 Times in 0 Posts
    1. This is scary. That defeats the entire purpose of local storage lol. Stupid apple

    2. That's why I made a library like this. The users didn't have to parse anything themselves. The library does it for them. Just trying to clean the library up with this shorthand stuff, but it won't affect the user of the lib.

    3. From what I remember reading using other domains to gain more storage was frowned on. I can see why. If I needed more space and was OK with going to the internet to get it (reaching out to other domains would require I assume hitting the internet), I'd just use ajax to send the data to be stored on the server.

    4. Would love to have an example of this. Guessing though this would trade-off performance since when reading/writing to localStorage we'd have to zip/unzip the data each time. Would have to test to see how much overhead it adds.
    Last edited by RickP; 11-28-2012 at 02:27 PM.

  • #11
    Senior Coder Dormilich's Avatar
    Join Date
    Jan 2010
    Location
    Behind the Wall
    Posts
    3,237
    Thanks
    12
    Thanked 340 Times in 336 Posts
    Quote Originally Posted by rnd me View Post
    i can post an inline zipper if there's interest. the deflate is ~10kb and the inflate is ~6kb; obviously those weights can MORE than pay for themselves pretty quickly.
    yepp, there’s interest.
    The computer is always right. The computer is always right. The computer is always right. Take it from someone who has programmed for over ten years: not once has the computational mechanism of the machine malfunctioned.
    André Behrens, NY Times Software Developer

  • #12
    New Coder
    Join Date
    Nov 2012
    Posts
    26
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Updated the library code on the first post.

    - Changed to stringifying an array of objects to local storage so parsing can also be done 100% by JSON

    - Added TruncateTable function

    - Added Update function

  • #13
    Senior Coder rnd me's Avatar
    Join Date
    Jun 2007
    Location
    Urbana
    Posts
    4,292
    Thanks
    10
    Thanked 583 Times in 564 Posts
    Quote Originally Posted by RickP View Post
    3. From what I remember reading using other domains to gain more storage was frowned on. I can see why. If I needed more space and was OK with going to the internet to get it (reaching out to other domains would require I assume hitting the internet), I'd just use ajax to send the data to be stored on the server.

    4. Would love to have an example of this. Guessing though this would trade-off performance since when reading/writing to localStorage we'd have to zip/unzip the data each time. Would have to test to see how much overhead it adds.
    3. the spec frowns on subdomains specifically. if you use a working manifest in that HTML page's <HTML> tag, you will not need the internet to access the domains localStorage. As long as you have enough access to another domain to publish an HTML file, you can embed that html file from another domain and talk to it using x-domain methods like iframe.src="#"+JSON.stringify(data) and top.postMessage(data).



    I am planning on making an abstracter that lets you list several domains, filling them as needed and keeping track of what's where to provide at least 50megs using the 10 or so domains i control.

    if i can get volunteers to embed more html pages in other domains, that amount can grow by leaps and bounds. the hosting cost would be minimal: no server processing, and ~3kb of bandwidth for a new visitor, no repeat traffic.

    Anyway, that's not done yet, but i'll throw what i have in the post a code section when it's ready. gimme a couple weeks; i'm swamped.


    4. you got it, gimme a few mins to find it.
    my site (updated 13/9/26)
    BROWSER STATS [% share] (2014/5/28) IE7:0.1, IE8:5.3, IE11:8.4, IE9:3.2, IE10:3.2, FF:18.2, CH:46, SF:7.9, NON-MOUSE:32%

  • #14
    New Coder
    Join Date
    Nov 2012
    Posts
    26
    Thanks
    1
    Thanked 0 Times in 0 Posts
    3. I could add this to my database library that I have here also. It would 100% abstract where the data is stored and retrieved from. I'm confused as to how this works though (again new to web development. normally doing console/desktop stuff). What is a working manifest int he <HTML> tag look like and what does it do? Why do you need volunteers to embed html pages in other domains? Not really following how this all works, but I like the idea of it. I do get the impression though that if this caught on it would be a reason for browsers to stop supporting it as it's sort of exploiting storage and it sounds like on the web side here so much of how to store data on the client is up in the air. Standards seem to many and people seem to be arguing over what's the best and what to support. Your Apple example is another example of browsers starting to rebel against local storage.

    4. Thanks. I would love to add this to my database library also to save even more space when storing the data.
    Last edited by RickP; 11-28-2012 at 03:20 PM.

  • #15
    Senior Coder rnd me's Avatar
    Join Date
    Jun 2007
    Location
    Urbana
    Posts
    4,292
    Thanks
    10
    Thanked 583 Times in 564 Posts

    deflate and inflate using javascript

    Intro
    By popular demand, here is how i "zip and unzip" strings in javascript. I did not write the orig, that was done a long time a go in a land far away. But, i have managed to squeeze 55kb into 8kb, and 20kb into 5kb, making the package useful in real-life. i made a peppering of other minor optimizations. My point is that even though i didn't come up with this, it's still as it stands below, my pride and joy. This was an ace up my sleeve that I decided to throw down on the table for all to see. X-mas comes early this year...





    Code


    Live Copy

    Code:
    /* Copyright (C) 1999 Masanao Izumo <iz@onicos.co.jp>
     * Version: 1.0.1
     * LastModified: Dec 25 1999.
     * updated and compressed by dandavis, 2012 (attribution only derivative)
     * Presumed abandonware, but retain this notice for public use.
     */
    
    // 5kb:
    var inflate=function(){function U(){this[J]=this[R]=null}function V(){this.n=this.b=this.e=0,this.t=null}function K(h,v,u,b,a,p){this[D]=16,this.N_MAX=288,this[z]=0,this[C]=null,this.m=0;var q=B(this[D]+1),m,c,l,g,d,e,f,i=B(this[D]+1),j,k,A,w=new V,r=B(this[D]);g=B(this.N_MAX);var x,s=B(this[D]+1),n,t,y;y=this[C]=null;for(d=0;d<q[E];d++)q[d]=0;for(d=0;d<i[E];d++)i[d]=0;for(d=0;d<r[E];d++)r[d]=null;for(d=0;d<g[E];d++)g[d]=0;for(d=0;d<s[E];d++)s[d]=0;m=256<v?h[256]:this[D],j=h,k=0,d=v;do q[j[k]]++,k++;while(0<--d);if(q[0]==v)this[C]=null,this[z]=this.m=0;else{for(e=1;e<=this[D]&&0==q[e];e++);f=e,p<e&&(p=e);for(d=this[D];0!=d&&0==q[d];d--);l=d,p>d&&(p=d);for(n=1<<e;e<d;e++,n<<=1)if(0>(n-=q[e])){this[z]=2,this.m=p;return}if(0>(n-=q[d]))this[z]=2,this.m=p;else{q[d]+=n,s[1]=e=0,j=q,k=1;for(A=2;0<--d;)s[A++]=e+=j[k++];j=h,d=k=0;do 0!=(e=j[k++])&&(g[s[e]++]=d);while(++d<v);v=s[l],s[0]=d=0,j=g,k=0,g=-1,x=i[0]=0,A=null;for(t=0;f<=l;f++)for(h=q[f];0<h--;){for(;f>x+i[1+g];){x+=i[1+g],g++,t=(t=l-x)>p?p:t;if((c=1<<(e=f-x))>h+1){c-=h+1;for(A=f;++e<t&&!((c<<=1)<=q[++A]);)c-=q[A]}x+e>m&&x<m&&(e=m-x),t=1<<e,i[1+g]=e,A=B(t);for(c=0;c<t;c++)A[c]=new V;null==y?y=this[C]=new U:y=y[R]=new U,y[R]=null,y[J]=A,r[g]=A,0<g&&(s[g]=d,w.b=i[g],w.e=16+e,w.t=A,e=(d&(1<<x)-1)>>x-i[g],r[g-1][e].e=w.e,r[g-1][e].b=w.b,r[g-1][e].n=w.n,r[g-1][e].t=w.t)}w.b=f-x,k>=v?w.e=99:j[k]<u?(w.e=256>j[k]?16:15,w.n=j[k++]):(w.e=a[j[k]-u],w.n=b[j[k++]-u]),c=1<<f-x;for(e=d>>x;e<t;e+=c)A[e].e=w.e,A[e].b=w.b,A[e].n=w.n,A[e].t=w.t;for(e=1<<f-1;0!=(d&e);e>>=1)d^=e;for(d^=e;(d&(1<<x)-1)!=s[g];)x-=i[g],g--}this.m=i[1],this[z]=0!=n&&1!=l?1:0}}}function f(h){for(;y<h;)H|=(M[E]==S?-1:M.charCodeAt(S++)&255)<<y,y+=8}function i(h){return H&ba[h]}function l(h){H>>=h,y-=h}function N(h,v,u){var b,a,p;if(0==u)return 0;for(p=0;;){f(r),a=s[J][i(r)];for(b=a.e;16<b;){if(99==b)return-1;l(a.b),b-=16,f(b),a=a.t[i(b)],b=a.e}l(a.b);if(16==b)j&=F-1,h[v+p++]=n[j++]=a.n;else{if(15==b)break;f(b),k=a.n+i(b),l(b),f(I),a=T[J][i(I)];for(b=a.e;16<b;){if(99==b)return-1;l(a.b),b-=16,f(b),a=a.t[i(b)],b=a.e}l(a.b),f(b),G=j-a.n-i(b);for(l(b);0<k&&p<u;)k--,G&=F-1,j&=F-1,h[v+p++]=n[j++]=n[G++]}if(p==u)return u}return t=-1,p}function ca(h,j,k){var b,a,p,q,m,c,n,g=B(316);for(b=0;b<g[E];b++)g[b]=0;f(5),c=257+i(5),l(5),f(5),n=1+i(5),l(5),f(4),b=4+i(4),l(4);if(286<c||30<n)return-1;for(a=0;a<b;a++)f(3),g[W[a]]=i(3),l(3);for(;19>a;a++)g[W[a]]=0;r=7,a=new K(g,19,19,null,null,r);if(0!=a[z])return-1;s=a[C],r=a.m,q=c+n;for(b=p=0;b<q;)if(f(r),m=s[J][i(r)],a=m.b,l(a),a=m.n,16>a)g[b++]=p=a;else if(16==a){f(2),a=3+i(2),l(2);if(b+a>q)return-1;for(;0<a--;)g[b++]=p}else{17==a?(f(3),a=3+i(3),l(3)):(f(7),a=11+i(7),l(7));if(b+a>q)return-1;for(;0<a--;)g[b++]=0;p=0}r=da,a=new K(g,c,257,X,Y,r),0==r&&(a[z]=1);if(0!=a[z])return 1==a[z],-1;s=a[C],r=a.m;for(b=0;b<n;b++)g[b]=g[b+c];return I=ea,a=new K(g,n,0,Z,$,I),T=a[C],I=a.m,0==I&&257<c?-1:(1==a[z],0!=a[z]?-1:N(h,j,k))}function fa(h,v,u){var b,a;for(b=0;b<u&&(!L||-1!=t);){if(0<k){if(t!=ga)for(;0<k&&b<u;)k--,G&=F-1,j&=F-1,h[v+b++]=n[j++]=n[G++];else{for(;0<k&&b<u;)k--,j&=F-1,f(8),h[v+b++]=n[j++]=i(8),l(8);0==k&&(t=-1)}if(b==u)break}if(-1==t){if(L)break;f(1),0!=i(1)&&(L=!0),l(1),f(2),t=i(2),l(2),s=null,k=0}switch(t){case 0:a=h;var p=v+b,q=u-b,m=void 0,m=y&7;l(m),f(16),m=i(16),l(16),f(16);if(m!=(~H&65535))a=-1;else{l(16),k=m;for(m=0;0<k&&m<q;)k--,j&=F-1,f(8),a[p+m++]=n[j++]=i(8),l(8);a=(0==k&&(t=-1),m)}break;case 1:if(null!=s)a=N(h,v+b,u-b);else a:{a=h,p=v+b,q=u-b;if(null==O){for(var c=void 0,m=B(288),c=0;144>c;c++)m[c]=8;for(;256>c;c++)m[c]=9;for(;280>c;c++)m[c]=7;for(;288>c;c++)m[c]=8;P=7,c=new K(m,288,257,X,Y,P);if(0!=c[z]){a=(alert("HufBuild error: "+c[z]),-1);break a}O=c[C],P=c.m;for(c=0;30>c;c++)m[c]=5;Q=5,c=new K(m,30,0,Z,$,Q);if(1<c[z]){a=(O=null,alert("HufBuild error: "+c[z]),-1);break a}aa=c[C],Q=c.m}a=(s=O,T=aa,r=P,I=Q,N(a,p,q))}break;case 2:null!=s?a=N(h,v+b,u-b):a=ca(h,v+b,u-b);break;default:a=-1}if(-1==a)return L?0:-1;b+=a}return b}var B=Array,z="status",E="length",C="root",D="BMAX",J="list",R="next",Q,F=32768,ga=0,da=9,ea=6,n,j,O=null,aa,P,H,y,t,L,k,G,s,T,r,I,M,S,ba=[0,1,3,7,15,31,63,127,255,511,1023,2047,4095,8191,16383,32767,65535],X=[3,4,5,6,7,8,9,10,11,13,15,17,19,23,27,31,35,43,51,59,67,83,99,115,131,163,195,227,258,0,0],Y=[0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0,99,99],Z=[1,2,3,4,5,7,9,13,17,25,33,49,65,97,129,193,257,385,513,769,1025,1537,2049,3073,4097,6145,8193,12289,16385,24577],$=[0,0,0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13],W=[16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15];return window.inflate=function(h){var f,i,b;null==n&&(n=B(2*F)),y=H=j=0,t=-1,L=!1,k=G=0,s=null,M=h,S=0,f=B(1024);for(h="";0<(i=fa(f,0,f[E]));)for(b=0;b<i;b++)h+=String.fromCharCode(f[b]);return M=null,h}}();
    
    
    // 8kb:
    var deflate=function(){var e,t,n,r,i=null,s,o,u,a,f,l,c,h,p,d,v,m,g,y,b,w,E,S,x,T,N,C,k,L,A,O,M,_,D,P,H,B,j,F,I,q,R,U,z,W,X,V,$,J,K,Q,G,Y,Z,et,tt,nt=function(){this.dl=this.fc=0},rt=function(){this.extra_bits=this.static_tree=this.dyn_tree=null,this.max_code=this.max_length=this.elems=this.extra_base=0},it=function(e,t,n,r){this.good_length=e,this.max_lazy=t,this.nice_length=n,this.max_chain=r},st=function(){this.next=null,this.len=0,this.ptr=[8192],this.off=0},ot=[0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0],ut=[0,0,0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13],at=[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,3,7],ft=[16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15],lt=[new it(0,0,0,0),new it(4,4,8,4),new it(4,5,16,8),new it(4,6,32,32),new it(4,4,16,16),new it(8,16,32,32),new it(8,16,128,128),new it(8,32,128,256),new it(32,128,258,1024),new it(32,258,258,4096)],ct=function(r){i[o+s++]=r;if(8192==o+s&&0!=s){var u,r=(null!=e?(u=e,e=e.next):u=new st,u.next=null,u.len=u.off=0,u);null==t?t=n=r:n=n.next=r,r.len=s-o;for(u=0;u<r.len;u++)r.ptr[u]=i[o+u];s=o=0}},ht=function(e){e&=65535,8190>o+s?(i[o+s++]=e&255,i[o+s++]=e>>>8):(ct(e&255),ct(e>>>8))},pt=function(){v=(v<<5^a[E+3-1]&255)&8191,m=c[32768+v],c[E&32767]=m,c[32768+v]=E},dt=function(e,t){Ot(t[e].fc,t[e].dl)},vt=function(e,t,n){return e[t].fc<e[n].fc||e[t].fc==e[n].fc&&R[t]<=R[n]},mt=function(e,t,n){var r;for(r=0;r<n&&tt<et.length;r++)e[t+r]=et.charCodeAt(tt++)&255;return r},gt=function(e){var t=N,n=E,r,i=w,s=32506<E?E-32506:0,o=E+258,u=a[n+i-1],f=a[n+i];w>=L&&(t>>=2);do if(r=e,a[r+i]==f&&a[r+i-1]==u&&a[r]==a[n]&&a[++r]==a[n+1]){n+=2,r++;do;while(a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&a[++n]==a[++r]&&n<o);r=258-(o-n),n=o-258;if(r>i){S=e,i=r;if(258<=r)break;u=a[n+i-1],f=a[n+i]}}while((e=c[e&32767])>s&&0!=--t);return i},yt=function(){var e,t,n=65536-T-E;if(-1==n)n--;else if(65274<=E){for(e=0;32768>e;e++)a[e]=a[e+32768];S-=32768,E-=32768,d-=32768;for(e=0;8192>e;e++)t=c[32768+e],c[32768+e]=32768<=t?t-32768:0;for(e=0;32768>e;e++)t=c[e],c[e]=32768<=t?t-32768:0;n+=32768}x||(e=mt(a,E+T,n),0>=e?x=!0:T+=e)},bt=function(e,n,i){var f;if(!r){if(!x){p=h=0;var l,F;if(0==_[0].dl){P.dyn_tree=A,P.static_tree=M,P.extra_bits=ot,P.extra_base=257,P.elems=286,P.max_length=15,P.max_code=0,H.dyn_tree=O,H.static_tree=_,H.extra_bits=ut,H.extra_base=0,H.elems=30,H.max_length=15,H.max_code=0,B.dyn_tree=D,B.static_tree=null,B.extra_bits=at,B.extra_base=0,B.elems=19,B.max_length=7;for(F=l=B.max_code=0;28>F;F++){W[F]=l;for(f=0;f<1<<ot[F];f++)U[l++]=F}U[l-1]=F;for(F=l=0;16>F;F++){X[F]=l;for(f=0;f<1<<ut[F];f++)z[l++]=F}for(l>>=7;30>F;F++){X[F]=l<<7;for(f=0;f<1<<ut[F]-7;f++)z[256+l++]=F}for(f=0;15>=f;f++)j[f]=0;for(f=0;143>=f;)M[f++].dl=8,j[8]++;for(;255>=f;)M[f++].dl=9,j[9]++;for(;279>=f;)M[f++].dl=7,j[7]++;for(;287>=f;)M[f++].dl=8,j[8]++;xt(M,287);for(f=0;30>f;f++)_[f].dl=5,_[f].fc=Mt(f,5);Et()}for(f=0;8192>f;f++)c[32768+f]=0;C=lt[k].max_lazy,L=lt[k].good_length,N=lt[k].max_chain,d=E=0,T=mt(a,0,65536);if(0>=T)x=!0,T=0;else{for(x=!1;262>T&&!x;)yt();for(f=v=0;2>f;f++)v=(v<<5^a[f]&255)&8191}t=null,o=s=0,3>=k?(w=2,b=0):(b=2,y=0),u=!1}r=!0;if(0==T)return u=!0,0}if((f=wt(e,n,i))==i)e=i;else if(u)e=f;else{if(3>=k)for(;0!=T&&null==t;){pt(),0!=m&&32506>=E-m&&(b=gt(m),b>T&&(b=T));if(3<=b)if(F=Lt(E-S,b-3),T-=b,b<=C){b--;do E++,pt();while(0!=--b);E++}else E+=b,b=0,v=a[E]&255,v=(v<<5^a[E+1]&255)&8191;else F=Lt(0,a[E]&255),T--,E++;for(F&&(kt(0),d=E);262>T&&!x;)yt()}else for(;0!=T&&null==t;){pt(),w=b,g=S,b=2,0!=m&&w<C&&32506>=E-m&&(b=gt(m),b>T&&(b=T),3==b&&4096<E-S&&b--);if(3<=w&&b<=w){F=Lt(E-1-g,w-3),T-=w-1,w-=2;do E++,pt();while(0!=--w);y=0,b=2,E++,F&&(kt(0),d=E)}else 0!=y?(Lt(0,a[E-1]&255)&&(kt(0),d=E),E++,T--):(y=1,E++,T--);for(;262>T&&!x;)yt()}e=(0==T&&(0!=y&&Lt(0,a[E-1]&255),kt(1),u=!0),f+wt(e,f+n,i-f))}return e},wt=function(n,r,u){var a,f,l;for(a=0;null!=t&&a<u;){f=u-a,f>t.len&&(f=t.len);for(l=0;l<f;l++)n[r+a+l]=t.ptr[t.off+l];t.off+=f,t.len-=f,a+=f,0==t.len&&(f=t,t=t.next,f.next=e,e=f)}if(a==u)return a;if(o<s){f=u-a,f>s-o&&(f=s-o);for(l=0;l<f;l++)n[r+a+l]=i[o+l];o+=f,a+=f,s==o&&(s=o=0)}return a},Et=function(){var e;for(e=0;286>e;e++)A[e].fc=0;for(e=0;30>e;e++)O[e].fc=0;for(e=0;19>e;e++)D[e].fc=0;A[256].fc=1,Q=$=J=K=Y=Z=0,G=1},St=function(e,t){for(var n=F[t],r=t<<1;r<=I;){r<I&&vt(e,F[r+1],F[r])&&r++;if(vt(e,n,F[r]))break;F[t]=F[r],t=r,r<<=1}F[t]=n},xt=function(e,t){var n=[16],r=0,i;for(i=1;15>=i;i++)r=r+j[i-1]<<1,n[i]=r;for(r=0;r<=t;r++)i=e[r].dl,0!=i&&(e[r].fc=Mt(n[i]++,i))},Tt=function(e){var t=e.dyn_tree,n=e.static_tree,r=e.elems,i,s=-1,o=r;I=0,q=573;for(i=0;i<r;i++)0!=t[i].fc?(F[++I]=s=i,R[i]=0):t[i].dl=0;for(;2>I;)i=F[++I]=2>s?++s:0,t[i].fc=1,R[i]=0,Y--,null!=n&&(Z-=n[i].dl);e.max_code=s;for(i=I>>1;1<=i;i--)St(t,i);do i=F[1],F[1]=F[I--],St(t,1),n=F[1],F[--q]=i,F[--q]=n,t[o].fc=t[i].fc+t[n].fc,R[i]>R[n]+1?R[o]=R[i]:R[o]=R[n]+1,t[i].dl=t[n].dl=o,F[1]=o++,St(t,1);while(2<=I);F[--q]=F[1],o=e.dyn_tree,i=e.extra_bits;var r=e.extra_base,n=e.max_code,u=e.max_length,a=e.static_tree,f,l,c,h,p=0;for(l=0;15>=l;l++)j[l]=0;o[F[q]].dl=0;for(e=q+1;573>e;e++)f=F[e],l=o[o[f].dl].dl+1,l>u&&(l=u,p++),o[f].dl=l,f>n||(j[l]++,c=0,f>=r&&(c=i[f-r]),h=o[f].fc,Y+=h*(l+c),null!=a&&(Z+=h*(a[f].dl+c)));if(0!=p){do{for(l=u-1;0==j[l];)l--;j[l]--,j[l+1]+=2,j[u]--,p-=2}while(0<p);for(l=u;0!=l;l--)for(f=j[l];0!=f;)i=F[--e],i>n||(o[i].dl!=l&&(Y+=(l-o[i].dl)*o[i].fc,o[i].fc=l),f--)}xt(t,s)},Nt=function(e,t){var n,r=-1,i,s=e[0].dl,o=0,u=7,a=4;0==s&&(u=138,a=3),e[t+1].dl=65535;for(n=0;n<=t;n++)i=s,s=e[n+1].dl,++o<u&&i==s||(o<a?D[i].fc+=o:0!=i?(i!=r&&D[i].fc++,D[16].fc++):10>=o?D[17].fc++:D[18].fc++,o=0,r=i,0==s?(u=138,a=3):i==s?(u=6,a=3):(u=7,a=4))},Ct=function(e,t){var n,r=-1,i,s=e[0].dl,o=0,u=7,a=4;0==s&&(u=138,a=3);for(n=0;n<=t;n++)if(i=s,s=e[n+1].dl,!(++o<u&&i==s)){if(o<a){do dt(i,D);while(0!=--o)}else 0!=i?(i!=r&&(dt(i,D),o--),dt(16,D),Ot(o-3,2)):10>=o?(dt(17,D),Ot(o-3,3)):(dt(18,D),Ot(o-11,7));o=0,r=i,0==s?(u=138,a=3):i==s?(u=6,a=3):(u=7,a=4)}},kt=function(e){var t,n,r;r=E-d,V[K]=Q,Tt(P),Tt(H);var i;Nt(A,P.max_code),Nt(O,H.max_code),Tt(B);for(i=18;3<=i&&0==D[ft[i]].dl;i--);i=(Y+=3*(i+1)+14,i),t=Y+3+7>>3,n=Z+3+7>>3,n<=t&&(t=n);if(r+4<=t&&0<=d){Ot(0+e,3),_t(),ht(r),ht(~r);for(i=0;i<r;i++)ct(a[d+i])}else if(n==t)Ot(2+e,3),At(M,_);else{Ot(4+e,3),r=P.max_code+1,t=H.max_code+1,i+=1,Ot(r-257,5),Ot(t-1,5),Ot(i-4,4);for(n=0;n<i;n++)Ot(D[ft[n]].dl,3);Ct(A,r-1),Ct(O,t-1),At(A,O)}Et(),0!=e&&_t()},Lt=function(e,t){l[$++]=t,0==e?A[t].fc++:(e--,A[U[t]+256+1].fc++,O[(256>e?z[e]:z[256+(e>>7)])&255].fc++,f[J++]=e,Q|=G),G<<=1,0==($&7)&&(V[K++]=Q,Q=0,G=1);if(2<k&&0==($&4095)){var n=8*$,r=E-d,i;for(i=0;30>i;i++)n+=O[i].fc*(5+ut[i]);n>>=3;if(J<parseInt($/2)&&n<parseInt(r/2))return!0}return 8191==$||8192==J},At=function(e,t){var n,r,i=0,s=0,o=0,u=0,a,c;if(0!=$)do 0==(i&7)&&(u=V[o++]),r=l[i++]&255,0==(u&1)?dt(r,e):(a=U[r],dt(a+256+1,e),c=ot[a],0!=c&&(r-=W[a],Ot(r,c)),n=f[s++],a=(256>n?z[n]:z[256+(n>>7)])&255,dt(a,t),c=ut[a],0!=c&&(n-=X[a],Ot(n,c))),u>>=1;while(i<$);dt(256,e)},Ot=function(e,t){p>16-t?(h|=e<<p,ht(h),h=e>>16-p,p+=t-16):(h|=e<<p,p+=t)},Mt=function(e,t){var n=0;do n|=e&1,e>>=1,n<<=1;while(0<--t);return n>>1},_t=function(){8<p?ht(h):0<p&&ct(h),p=h=0};return window.deflate=function(s,o){var u,h;et=s,tt=0,"undefined"==typeof o&&(o=6),(u=o)?1>u?u=1:9<u&&(u=9):u=6,k=u,x=r=!1;if(null==i){e=t=n=null,i=[8192],a=[65536],f=[8192],l=[32832],c=Array(65536),A=[573];for(u=0;573>u;u++)A[u]=new nt;O=[61];for(u=0;61>u;u++)O[u]=new nt;M=[288];for(u=0;288>u;u++)M[u]=new nt;_=[30];for(u=0;30>u;u++)_[u]=new nt;D=[39];for(u=0;39>u;u++)D[u]=new nt;P=new rt,H=new rt,B=new rt,j=[16],F=[573],R=[573],U=Array(256),z=[512],W=[29],X=[30],V=Array(1024)}for(var p=[1024],d=[];0<(u=bt(p,0,p.length));){var v=[u];for(h=0;h<u;h++)v[h]=String.fromCharCode(p[h]);d[d.length]=v.join("")}return et=null,d.join("")}}()



    Considerations
    Note that some char codes cannot be zipped. I am not 100% sure why this is. ascii is 100% ok, but SOME intl char sometimes throw it. forgive me for not figuring this out in detail, feel free, but the issue is easy to work around using escape/unescape or window.btoa/atob where available/possible. I've never had an issue using it since i figured out that much about the problem.


    if you are storing plain text and json, you're fine.
    binary and intl chars will need dumbed-down before zipping.
    Even in that case, even with the escape overhead, zipping still saves a lot of space; escape only fattens the zip ~10-20%.

    here is a usage example, proof of operation, and a demo of the intl complication:


    Usage
    Code:
    // run in firebug/devtools/"F12" on http://www.codingforums.com/showthread.php?t=283110
    function testZip(str){
      return inflate(deflate(str))==str;
    }
    
    testZip(document.head.outerHTML); // true
    testZip(document.body.outerHTML); // false
    testZip(escape(document.body.outerHTML)); // true
    deflate() is akin to escape() and inflate() is akin to unescape().


    Performance


    compression:
    • BYTES string source // note
    • 83,816 document.body.outerHTML.length // orig
    • 18,941 deflate(document.body.outerHTML).length // (non-operational)
    • 21,406 deflate(escape(document.body.outerHTML)).length //works


    that's about 4:1 on fairly non-repetitive HTML. JSON usually give me 5:1, and HTML tables can go up to 20:1. Elegant plain text is lucky to get 3:1. You can expect at least a doubling of the amount you can store, so 10mb is the new 5mb.

    CPU usage:
    my timings on 30,000 bytes of pretty repetitive json, running on a cheap year-old desktop:
    • 0.09ms deflate (30kb _> 1.3kb)
    • 0.13ms inflate (1.3kb _> 30kb)




    notes


    even on a phone 1/100th the speed of my desktop, we are talking about 13ms to inflate the data. the iphone 4S tests at about 1/13th of my desktop for comparison. so, on the iphone, this should happen in ~2ms. jquery.js might take 10ms on that device, not a humanly-perceivable delay.


    bigger payloads will take more time, in a linear fashion, but they will also compresses with more efficiency. 105kb when compressed is more than 5kb smaller than 100kb when compressed. in fact, it would likely add less than 5% to the compressed size because of existing repetition.


    breaking the data into chunks can reduce CPU at the cost of disk.
    it WILL take a noticeable while to deflate 5mb of text on a smartphone. If you are trying to do this often, or at a bad time, say, onkeypress, your app will grind to a halt. If you had 10 different 500kb packages, you would only have to touch ~1/10th the data, and your pauses would be 10th as long.



    you can use deflate() solely in your build process and tuck inflate() into your distribute-able package for smaller downloads.


    usually the additional inflate CPU overhead is less time than downloading the extra uncompressed data would have been. This remains true on "under-powered" smartphones because their network, even at 4g is slow.


    keep in mind that shipping a zipped package will reduce the efficiency of server-based transparent gzip transfer encoding. This is for (hopefully) obvious reasons. it's not like it hurts, you're still saving a ton compared to the orig. the zip/zip overhead is only about 10%, and given the advantage of being able to persist the compressed version client side, i think it's almost always worth it. a lot of servers don't even gzip or cache JS files, so that is not an issue for many.


    if you server doesn't support gzip at all, you can probably get better app loading perf by zipping all your CSS files into one string, and using a javascript "addCSSasString()" type method. same benefits for JS.



    one cool thing is that is when you ship, say jQuery.js as a zipped string, you can save the whole lib to localStorage. with a conditional adder like yepnope or whatever, you can load jquery locally upon the next visit instead of pinging the server to check the HTTP cache status, or even downloading it each time on a non-caching server. i've measured many a big improvement on production sites where i've rolled the pattern out, despite a few naysayers slamming localStorage performance a couple years ago...

    here is an example of deploying code like that: http://danml.com/js/?compression,mode=json
    as opposed to http://danml.com/js/compression.js or http://danml.com/js/?compression

    to use that url's specific functionality, you have to define App and App.X, but then, any and all externally-loaded code is available in App.X, which is one JSON.stringify() away from persistence.
    you should modify you own code server to dish out the callbacks or containers that your projects use.



    In conclusion, enjoy and let me know what you think of it.
    Last edited by rnd me; 11-28-2012 at 07:16 PM.
    my site (updated 13/9/26)
    BROWSER STATS [% share] (2014/5/28) IE7:0.1, IE8:5.3, IE11:8.4, IE9:3.2, IE10:3.2, FF:18.2, CH:46, SF:7.9, NON-MOUSE:32%

  • Users who have thanked rnd me for this post:

    RickP (11-28-2012)


  •  
    Page 1 of 2 12 LastLast

    Tags for this Thread

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •