View Full Version : Parsing CSV (semi-advanced)

05-01-2008, 05:44 PM
Hey all, my current project is a fairly big site with a group of coders, and we are all trying to determine the fastest way to bring in lots of data from a database.

We have all the back end set up and found it most efficient to send through the information in CSV format. When it got to the front end we simply used javascripts split() function to break up the values and then we need to plug those values into another object we created.

So to give you an idea of what we are doing...
We are creating a service to help people connect with landlords. To manage all the possible houses, we have created a house object which contains an array of values (low end of rent, high end of rent, address, etc...). Once the object is set up, you can access these values by typing house[37].val["address"] and it holds "123 any street" for example.

To set these values you access the addVal function. So to set the address, you would type
house[37].addVal("address", "123 any street");
Example CSV that we have available in a string is...
id:lat:lng:rent_low:rent_high:address:::541:43.4651:-80.5623:125:130:276 Beechlawn Drive:::542:43.4605:-80.5342:150:490:191 Erb St W :::571:43.457:-80.5533:200:220:402 Midwood Crescent:::
The different houses are seperated by three :'s (:::) and the values by a single colon. The first house's values are mearly the names of the columns in the database and how the values will be referenced from that point on.

Finally, our current parsing code...

var cut = CSV.split(':::'); //cut now holds a string for every house
var n = 0; //A simple counter
for(i=0;i<cut.length;i++) //Iterate through all houses
var cut2=cut[i].split(':'); //cut2 now holds each value for a house in
//an array
if(i == 0){ //If the first house, set up an array of the
//field headings (["id", "rent_low" ... ])
var titles = cut2;
} else {
house[cut2[0]] = new objHouse(); //Create the new house object, put it
//in an array and reference it by id
n = 0;

//For each value in the cut2 array, add the value into the new house
//object, giving the title information and the value itself.
for each (item in cut2){
house[cut2[0]].addVal(titles[n], item);
So once again I pose the question: is there a faster way to accomplish this task? I recently just recoded everything from XML to CSV and gained a performance boost from a 15.5 second load time to a 6.7 second load time (remember, we are working with hundreds of houses and many more values (parking included or not, landlord contact information, etc) so the code needs to be as efficient as possible.

I am not asking anyone to dedicate hours of their time to try to speed this up, if we needed that we would pay someone to do it. I'm just asking as one coder to another, is there anything about this current method that jumps out at you as slow or could be improved.

Thanks all, while waiting I'll help out some other coders.