View Full Version : Interacting with another site... cURL?

11-24-2010, 10:41 PM
Hi all. Hypothetical scenario... let's say there's an online vendor and this vendor sells unique products (as in one of this, one of that, etc...). If I wanted to monitor their site for inventory updates then buy certain items as soon as they become available, is this something that could be feasible using cURL?

The assumptions are:
- I know roughly what time the inventory will be updated each day
- I have a way to identify each item of interest and its 'add to cart' link
- I have an account with the vendor

I have never used cURL but it was suggested that I should look into it if I wanted to attempt this. The goal is a sort of "snipe" program that would give the end user the best chance of successfully purchasing any number of unique items that other buyers will be eagerly wanting to purchase as well.

I'm not looking for a handholding on development, just whether or not this is feasible with cURL and/or if there are better tools that could be put to work here?

Thanks and happy Turkey Day!

11-25-2010, 02:23 AM
it is feasible. It's not a project I would look forward to working on, but I would probably use cURL to pull in the relevant HTML, then use simplehtmldom (http://simplehtmldom.sourceforge.net/) to parse the HTML to get what I need. However, doing it that way gives you no guarantees that your code won't break completely if they change their HTML.


12-01-2010, 07:57 PM
Jim, thanks for the link on simplehtmldom... that should be useful. I haven't found any better way of going about this, it's a little off the beaten path for me but I might give it a shot.

You're correct though that changes to the HTML on the vendor site could potentially break the app, I don't think there's any way around having to roll out updates to keep up.

12-01-2010, 08:09 PM
You *should* be able to do it, and if done intelligently enough, it *might* survive a few HTML tweaks.

I've done similar before (parsing and submitting from a script) and it is something I would look forward to doing. It's just not something I'd do for free ;)

cURL should handle the SSL connection and cookies just fine, and those are pretty much the pre-requisites for purchasing online. It's just a case of parsing the HTML and submitting the right data through cURL.

12-03-2010, 05:51 PM
Thanks Lamped, your input is appreciated.

I've been mulling this over as I work on other things. One potential pitfall I can see is that all of the cURL requests will be coming from one server (if this runs as a service via centralized software). This would make it fairly easy for the vendor to defeat the software should they decided they don't like what's going on.

The only way I can see around that would be to write a sort of client application using dotnet etc... with a web component containing all of the scripting needed. This would require that a web server be installed and running on the client end. I'm not sure how feasible that is. In theory I suppose it could work, presenting a different IP from each user. This would make it nearly impossible to defeat based on IP alone... unless said vendor decides to start black listing a bunch of clients (possible, but not a concern right now).

I can barely manage to get Apache and PHP installed on my own development machine most of the time, I'm not sure I could successfully build an install package to automate all of that! :) This might be more like a pipe dream.

Maybe ASP or ASP.net would be a better solution if something like a client is required? I believe there is a cURL equivalent from M$.

12-03-2010, 09:43 PM
I've written a client app in PHP using GTK, though I wouldn't recommend it for the weak-hearted. You're probably on the right track with C# as I hear it makes stuff like this relatively easy.

I'm a little concerned you mentioning the vendor may not like this. Perhaps you should talk to them?

12-04-2010, 02:39 AM
I think that may be the best route too. I did a little research and it appears that dotnet has api's that could do this.

I don't feel like this is something unethical, but if/when it becomes publicly available I can see where some clients of the vendor may not want to pay (me) a small fee to use the software and may be unhappy that others are gaining an advantage. I'm sure the vendor won't want unhappy customers so they may not be too keen on this. I think I'll wait until something is actually developed and at least tested before opening any lines of communication. It could be that the vendor ends up having an interest in the software, but I doubt it.