View Full Version : how does sites that do price comparison's get the info ?

03-17-2008, 03:34 PM
Hello all
how does sites that do price comparison get there info about the prices and such?
does they use the sites web services api ? or they use web spiders?

03-17-2008, 09:32 PM
Some do "scraping" ... which isn't ethical...
Others probably subscribe to some database(s) which provide the data?

If a site offers an RSS feed, that would be an ethical way to get the data.

I would guess that the type of website (what they sell) would be
a key thing on what is available.

You can do a search on Google for "web page scraping" to see how you
can make your own spider. Like I said before, it's not really ethical to steal
bandwidth from another site without permission.

03-17-2008, 09:50 PM
You just sign up for multiple datafeeds and integrate them into your system. For example, if you sign up for...


You're going to have data for any product from at least 3, and probably dozens of sites.

The tricky part is setting up the database to handle that much data.

I've already integrated amazon into one of my sites (http://www.yourlistonline.com) and plan on pulling in buy.com and shopzilla in the near future.