What if you used cURL, or file_get_contents to DL the files to a folder? Would that be much faster? Then you could parse them by whatever means works best. You have the BASH shell and any number of ways of doing it, that are probably much faster than SimpleXML, even if you just used them to search for relevant files and let SimpleXML do the heavy work, it might save a great deal of time. Plus the shell doesn't have a timeout limit.
umm yes, I was just now googling bash scripts to run a list of php files in a single cron job
Originally Posted by DrDOS
my only objections to bash and cURL is I've never used it, oh well, I think you're right, one of those to assemble all the seperate pages as single file and then parse
'no timeout limit' sounds lovely