Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 7 of 7
  1. #1
    Regular Coder
    Join Date
    Jul 2003
    Posts
    262
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Interacting with another site... cURL?

    Hi all. Hypothetical scenario... let's say there's an online vendor and this vendor sells unique products (as in one of this, one of that, etc...). If I wanted to monitor their site for inventory updates then buy certain items as soon as they become available, is this something that could be feasible using cURL?

    The assumptions are:
    - I know roughly what time the inventory will be updated each day
    - I have a way to identify each item of interest and its 'add to cart' link
    - I have an account with the vendor

    I have never used cURL but it was suggested that I should look into it if I wanted to attempt this. The goal is a sort of "snipe" program that would give the end user the best chance of successfully purchasing any number of unique items that other buyers will be eagerly wanting to purchase as well.

    I'm not looking for a handholding on development, just whether or not this is feasible with cURL and/or if there are better tools that could be put to work here?

    Thanks and happy Turkey Day!

  • #2
    New Coder
    Join Date
    Nov 2010
    Posts
    12
    Thanks
    0
    Thanked 1 Time in 1 Post
    it is feasible. It's not a project I would look forward to working on, but I would probably use cURL to pull in the relevant HTML, then use simplehtmldom to parse the HTML to get what I need. However, doing it that way gives you no guarantees that your code won't break completely if they change their HTML.

    -Jim
    Last edited by WA; 11-25-2010 at 04:55 AM.

  • #3
    Regular Coder
    Join Date
    Jul 2003
    Posts
    262
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Jim, thanks for the link on simplehtmldom... that should be useful. I haven't found any better way of going about this, it's a little off the beaten path for me but I might give it a shot.

    You're correct though that changes to the HTML on the vendor site could potentially break the app, I don't think there's any way around having to roll out updates to keep up.

  • #4
    Super Moderator
    Join Date
    Feb 2009
    Location
    England
    Posts
    539
    Thanks
    8
    Thanked 63 Times in 54 Posts
    You *should* be able to do it, and if done intelligently enough, it *might* survive a few HTML tweaks.

    I've done similar before (parsing and submitting from a script) and it is something I would look forward to doing. It's just not something I'd do for free

    cURL should handle the SSL connection and cookies just fine, and those are pretty much the pre-requisites for purchasing online. It's just a case of parsing the HTML and submitting the right data through cURL.
    lamped.co.uk :: Design, Development & Hosting
    marcgray.co.uk :: Technical blog

  • #5
    Regular Coder
    Join Date
    Jul 2003
    Posts
    262
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Thanks Lamped, your input is appreciated.

    I've been mulling this over as I work on other things. One potential pitfall I can see is that all of the cURL requests will be coming from one server (if this runs as a service via centralized software). This would make it fairly easy for the vendor to defeat the software should they decided they don't like what's going on.

    The only way I can see around that would be to write a sort of client application using dotnet etc... with a web component containing all of the scripting needed. This would require that a web server be installed and running on the client end. I'm not sure how feasible that is. In theory I suppose it could work, presenting a different IP from each user. This would make it nearly impossible to defeat based on IP alone... unless said vendor decides to start black listing a bunch of clients (possible, but not a concern right now).

    I can barely manage to get Apache and PHP installed on my own development machine most of the time, I'm not sure I could successfully build an install package to automate all of that! This might be more like a pipe dream.

    Maybe ASP or ASP.net would be a better solution if something like a client is required? I believe there is a cURL equivalent from M$.

  • #6
    Super Moderator
    Join Date
    Feb 2009
    Location
    England
    Posts
    539
    Thanks
    8
    Thanked 63 Times in 54 Posts
    I've written a client app in PHP using GTK, though I wouldn't recommend it for the weak-hearted. You're probably on the right track with C# as I hear it makes stuff like this relatively easy.

    I'm a little concerned you mentioning the vendor may not like this. Perhaps you should talk to them?
    lamped.co.uk :: Design, Development & Hosting
    marcgray.co.uk :: Technical blog

  • #7
    Regular Coder
    Join Date
    Jul 2003
    Posts
    262
    Thanks
    1
    Thanked 0 Times in 0 Posts
    I think that may be the best route too. I did a little research and it appears that dotnet has api's that could do this.

    I don't feel like this is something unethical, but if/when it becomes publicly available I can see where some clients of the vendor may not want to pay (me) a small fee to use the software and may be unhappy that others are gaining an advantage. I'm sure the vendor won't want unhappy customers so they may not be too keen on this. I think I'll wait until something is actually developed and at least tested before opening any lines of communication. It could be that the vendor ends up having an interest in the software, but I doubt it.


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •