I'm new to javascript but not programming. I recently learned about local storage and love the idea of it. I'm creating a small website that let's me track my golf scores and other stats and I'm doing it 100% on the client side and storing the results in the local storage.
I work in databases all day long so I love that style more than the key/value that local storage offers so I set out to make a small library that converts local storage into a more DB structure. Thought I'd share with others to see what their thoughts are and maybe it'll be useful for someone.
- The table name is a key in the local storage
- The data is stored as an array of stringified JSON per table (key)
- The data is compressed and uncompressed when written to and read from local storage
- The first entry in an item is the table definition
Below is the usage with comments:
Code:
<script language="javascript" type="text/javascript">
$(document).ready(OnDocumentReady);
function OnDocumentReady() {
// create a database object which allows many database operations
var db = new LocalStorageDB();
// this is just for testing to clear the local storage while we test
db.TruncateAll();
// deletes the data in the players table
db.TruncateTable("players");
// check if the table exists
if (!db.TableExists("players")) {
// create a new table
var tblPlayers = new LocalStorageTable("players");
// add columns to the new table
tblPlayers.AddColumn("name");
tblPlayers.AddColumn("email");
// add the table to the database
db.AddTable(tblPlayers);
}
// insert some records
// first parameter is the table name to insert into
// second parameter is an object that has the same fields as the table definition
db.Insert("players", { name: "Bob", email: "bob@gmail.com" });
db.Insert("players", { name: "Joe", email: "joe@gmail.com" });
db.Insert("players", { name: "Rick", email: "rick@gmail.com" });
db.Insert("players", { name: "Mike", email: "mike@gmail.com" });
db.Insert("players", { name: "Kevan", email: "kevan@gmail.com" });
// this will update the name to Ricky where the name is currently Rick
// first param is table name
// second param is object of the fields you want to update with the value you want to update to
// third param is the where clause. a function that when the condition returns true, will update those records
db.Update("players", { name: "Ricky" }, function (v) { return v.name == "Rick"; });
// deletes when returns true in function
// first parameter is table name
// second parameter is the where clause. a function in where if the condition returns true the record will be deleted
db.Delete("players", function (v) { return v.name == "Mike"; });
// query recrods
// first param is the table name
// second param is the where clause. a function that when the condition returns true, will return that record as part of an array
var rsNames = db.Query("players", function (v) { return v.name == "Rick" && v.email == "rick@gmail.com"; });
// the result is an array of objects that match the table structure. check it's length to determine if anything was returned. access it's
// records via an index, and it's columns via the field name defined when the table was created
if (rsNames.length > 0)
var email = rsNames[0].email;
// erase the table completely. records and structure
db.DropTable("players");
}
</script>
Looks great to work with. But just to keep in mind the browser memory (storage size) available to implement local JavaScript database. Below is the spec -
But it has to be persistent. To do that I'm using local storage and local storage can only store key/value as strings not objects. So at some level this has to be stored as a string and split to pull the data out that we want. I do plan on doing what you are saying and storing it as a string so that JSON.parse() on the entire string will make an array of objects for me and save me from having to call split() and loop, but I have to imagine at some level JSON.parse() is doing some kind of splitting and looping somehow on it's own.
Looks great to work with. But just to keep in mind the browser memory (storage size) available to implement local JavaScript database. Below is the spec -
Yeah I noticed this. My first intention of usage for this is for a golf score mobile website so the data is going to be very small. I want to use local storage because some golf courses are way out of the way and don't get great reception, so my golf mobile website is going to be 1 page that loads all it needs to, and never requires internet access outside of navigating to it first which a user could do when internet coverage is available and just keep the browser open for using it where coverage isn't available. I do then plan on having an option for them to upload the data to a server if they wish.
I think 5MB is a decent amount of memory when dealing with just strings. You can fit a ton of information into 5 MB of text.
that’s the whole point. why splitting yourself if JSON.parse() (resp. JSON.stringify()) already does the hard work for you?
Yeah, I agree. It does save my library from having to do heavy lifting. I'm new to javascript so I wasn't aware JSON.parse() was able to do the whole array parsing thing. I plan on storing it as comma separated JSON value string in localStorage now (instead of /r) and when I need to manipulate the data I'm going to prefix & append to a var string the brackets and pass that result to JSON.parse() to make my life easier.
Code:
var data = localStorage.getItem(tblName);
var objData = JSON.parse("[" + data + "]");
I think this would be easier than storing the brackets inside the string when having to insert new rows. I can just put a comma at the end and insert vs having to worry about that last bracket.
[EDIT]
Oh JSON.stringify() will also do this. Yeah I'll just use a temp array when inserting and pulling data out. Thanks for the tip!
When I parse the local storage stored as array of JSON objects I don't get an array of objects, I get an array of JSON strings. Is that what I'm supposed to get or should I get an array of objects that the JSON represents?
I do notice now I have / in the JSON string where before I didn't.
When I JSON.parse the local storage for a table the result is:
[{"name":"name"},{"name":"Rick"}]
But that's just an array of JSON string values. Is that what it should do? That would mean I have to JSON.parse those too to get objects. I thought 1 JSON.parse() would do that automatically for me?
1. this is wrong. iphone, in an attempt to promote it's $$$ Apps over webapps, clears the contents of localStorage when the browser restarts or when the phone crashes or is turned off. don't read about it, test it.
2. OP: obviously, fellow coders want you to work JSON.stringify and JSON.parse into the routines for your DB so they don't have to call them each time.
3. you can use another domain to get another 5MB. This requires converting your code to an async version, but doing so provides obvious benefits. I can post a x-domain localStorage proxy if anyone's interested. it uses postMessage() and hashChange() to communicate with a hidden iframe.
4. if dealing with text, you can zip the code before saving it to localstorage. This helps a lot because, for example, Chrome's localStrorage is utf16, which means two bytes per char; a 50% waste for most JSON. By using zip, you can make better use of the full 16bits, not to mention the space savings from deflate. i can post an inline zipper if there's interest. the deflate is ~10kb and the inflate is ~6kb; obviously those weights can MORE than pay for themselves pretty quickly.
__________________ my site (updated 5/13) STATS (2013/5) HTML5:90.2% MOB:14% IE7:0.5% IE8:8.6% IE9:9.8% IE10:10%
1. This is scary. That defeats the entire purpose of local storage lol. Stupid apple
2. That's why I made a library like this. The users didn't have to parse anything themselves. The library does it for them. Just trying to clean the library up with this shorthand stuff, but it won't affect the user of the lib.
3. From what I remember reading using other domains to gain more storage was frowned on. I can see why. If I needed more space and was OK with going to the internet to get it (reaching out to other domains would require I assume hitting the internet), I'd just use ajax to send the data to be stored on the server.
4. Would love to have an example of this. Guessing though this would trade-off performance since when reading/writing to localStorage we'd have to zip/unzip the data each time. Would have to test to see how much overhead it adds.
i can post an inline zipper if there's interest. the deflate is ~10kb and the inflate is ~6kb; obviously those weights can MORE than pay for themselves pretty quickly.
yepp, there’s interest.
__________________
please post your code wrapped in [CODE] [/CODE] tags
3. From what I remember reading using other domains to gain more storage was frowned on. I can see why. If I needed more space and was OK with going to the internet to get it (reaching out to other domains would require I assume hitting the internet), I'd just use ajax to send the data to be stored on the server.
4. Would love to have an example of this. Guessing though this would trade-off performance since when reading/writing to localStorage we'd have to zip/unzip the data each time. Would have to test to see how much overhead it adds.
3. the spec frowns on subdomains specifically. if you use a working manifest in that HTML page's <HTML> tag, you will not need the internet to access the domains localStorage. As long as you have enough access to another domain to publish an HTML file, you can embed that html file from another domain and talk to it using x-domain methods like iframe.src="#"+JSON.stringify(data) and top.postMessage(data).
I am planning on making an abstracter that lets you list several domains, filling them as needed and keeping track of what's where to provide at least 50megs using the 10 or so domains i control.
if i can get volunteers to embed more html pages in other domains, that amount can grow by leaps and bounds. the hosting cost would be minimal: no server processing, and ~3kb of bandwidth for a new visitor, no repeat traffic.
Anyway, that's not done yet, but i'll throw what i have in the post a code section when it's ready. gimme a couple weeks; i'm swamped.
4. you got it, gimme a few mins to find it.
__________________ my site (updated 5/13) STATS (2013/5) HTML5:90.2% MOB:14% IE7:0.5% IE8:8.6% IE9:9.8% IE10:10%
3. I could add this to my database library that I have here also. It would 100% abstract where the data is stored and retrieved from. I'm confused as to how this works though (again new to web development. normally doing console/desktop stuff). What is a working manifest int he <HTML> tag look like and what does it do? Why do you need volunteers to embed html pages in other domains? Not really following how this all works, but I like the idea of it. I do get the impression though that if this caught on it would be a reason for browsers to stop supporting it as it's sort of exploiting storage and it sounds like on the web side here so much of how to store data on the client is up in the air. Standards seem to many and people seem to be arguing over what's the best and what to support. Your Apple example is another example of browsers starting to rebel against local storage.
4. Thanks. I would love to add this to my database library also to save even more space when storing the data.
Intro
By popular demand, here is how i "zip and unzip" strings in javascript. I did not write the orig, that was done a long time a go in a land far away. But, i have managed to squeeze 55kb into 8kb, and 20kb into 5kb, making the package useful in real-life. i made a peppering of other minor optimizations. My point is that even though i didn't come up with this, it's still as it stands below, my pride and joy. This was an ace up my sleeve that I decided to throw down on the table for all to see. X-mas comes early this year...
Considerations
Note that some char codes cannot be zipped. I am not 100% sure why this is. ascii is 100% ok, but SOME intl char sometimes throw it. forgive me for not figuring this out in detail, feel free, but the issue is easy to work around using escape/unescape or window.btoa/atob where available/possible. I've never had an issue using it since i figured out that much about the problem.
if you are storing plain text and json, you're fine.
binary and intl chars will need dumbed-down before zipping.
Even in that case, even with the escape overhead, zipping still saves a lot of space; escape only fattens the zip ~10-20%.
here is a usage example, proof of operation, and a demo of the intl complication:
Usage
Code:
// run in firebug/devtools/"F12" on http://www.codingforums.com/showthread.php?t=283110
function testZip(str){
return inflate(deflate(str))==str;
}
testZip(document.head.outerHTML); // true
testZip(document.body.outerHTML); // false
testZip(escape(document.body.outerHTML)); // true
deflate() is akin to escape() and inflate() is akin to unescape().
that's about 4:1 on fairly non-repetitive HTML. JSON usually give me 5:1, and HTML tables can go up to 20:1. Elegant plain text is lucky to get 3:1. You can expect at least a doubling of the amount you can store, so 10mb is the new 5mb.
CPU usage:
my timings on 30,000 bytes of pretty repetitive json, running on a cheap year-old desktop:
0.09ms deflate (30kb _> 1.3kb)
0.13ms inflate (1.3kb _> 30kb)
notes
even on a phone 1/100th the speed of my desktop, we are talking about 13ms to inflate the data. the iphone 4S tests at about 1/13th of my desktop for comparison. so, on the iphone, this should happen in ~2ms. jquery.js might take 10ms on that device, not a humanly-perceivable delay.
bigger payloads will take more time, in a linear fashion, but they will also compresses with more efficiency. 105kb when compressed is more than 5kb smaller than 100kb when compressed. in fact, it would likely add less than 5% to the compressed size because of existing repetition.
breaking the data into chunks can reduce CPU at the cost of disk.
it WILL take a noticeable while to deflate 5mb of text on a smartphone. If you are trying to do this often, or at a bad time, say, onkeypress, your app will grind to a halt. If you had 10 different 500kb packages, you would only have to touch ~1/10th the data, and your pauses would be 10th as long.
you can use deflate() solely in your build process and tuck inflate() into your distribute-able package for smaller downloads.
usually the additional inflate CPU overhead is less time than downloading the extra uncompressed data would have been. This remains true on "under-powered" smartphones because their network, even at 4g is slow.
keep in mind that shipping a zipped package will reduce the efficiency of server-based transparent gzip transfer encoding. This is for (hopefully) obvious reasons. it's not like it hurts, you're still saving a ton compared to the orig. the zip/zip overhead is only about 10%, and given the advantage of being able to persist the compressed version client side, i think it's almost always worth it. a lot of servers don't even gzip or cache JS files, so that is not an issue for many.
if you server doesn't support gzip at all, you can probably get better app loading perf by zipping all your CSS files into one string, and using a javascript "addCSSasString()" type method. same benefits for JS.
one cool thing is that is when you ship, say jQuery.js as a zipped string, you can save the whole lib to localStorage. with a conditional adder like yepnope or whatever, you can load jquery locally upon the next visit instead of pinging the server to check the HTTP cache status, or even downloading it each time on a non-caching server. i've measured many a big improvement on production sites where i've rolled the pattern out, despite a few naysayers slamming localStorage performance a couple years ago...
to use that url's specific functionality, you have to define App and App.X, but then, any and all externally-loaded code is available in App.X, which is one JSON.stringify() away from persistence.
you should modify you own code server to dish out the callbacks or containers that your projects use.
In conclusion, enjoy and let me know what you think of it.
__________________ my site (updated 5/13) STATS (2013/5) HTML5:90.2% MOB:14% IE7:0.5% IE8:8.6% IE9:9.8% IE10:10%
Before you post, read our: 
Local Storage DB
