Originally Posted by Tynan
I'm surprised at how slow this script is, I actually have 2,500 pages to parse and at the moment I can't do 80. I'm happy to run this one a night in the middle of the night but is there another way to tackle this, perhaps i can simply retrieve the info and save it locally first, I assume the parsing would be lightening fast and the delay is in loading the remote page, in which case this is always going to be a problem
You're correct here, the parsing is going to be lightning fast as compared to the http request. What you'd like is to be able to request all the data at once, rather than having to make 2500 individual requests. Maybe there's a different url syntax that can be used to get this?
FWIW, I ran your script up to 120 requests, and all of them come in about 1.5 seconds apart. No 80s timeout for me, though I'm not on your server. But that means the 80s issue is likely *your* server, not the remote server.