jaywhy13
08-12-2009, 05:06 PM
Hi all, is there a tangible way to measure the benefit of reading the contents of a file from memory vs reading the file straight from the disk?
I have a LARGE csv file that I need to read data from and perform operations on it. I'm wondering if it would be best to read in the entire file and then do what I want to do from memory or just read straight from the file. I'm wondering what are the pro's and con's.
So far I'm reasoning it out... the question is
Is
time to read file into memory + n * time to process each row from memory
greater or less than
time taken to open file + n * time to process each in file
Please add some insight. I might need to also read only parts of the file into memory perhaps? I'm gunning at speed here.
I have a LARGE csv file that I need to read data from and perform operations on it. I'm wondering if it would be best to read in the entire file and then do what I want to do from memory or just read straight from the file. I'm wondering what are the pro's and con's.
So far I'm reasoning it out... the question is
Is
time to read file into memory + n * time to process each row from memory
greater or less than
time taken to open file + n * time to process each in file
Please add some insight. I might need to also read only parts of the file into memory perhaps? I'm gunning at speed here.