View Full Version : How do I perform automated search on a website?

08-11-2010, 09:52 PM
There is a website where people can check the status of their application for a certain service using a confirmation number. This is a basic website for just this purpose, so it is not keyword-searchable. Is it possible to write some kind of script that will check the status of the application for a range of confirmation codes and report the confirmation codes for which the status page contain a certain keyword?

08-11-2010, 11:06 PM
Would need to look at the site.

08-12-2010, 03:56 AM
It's possible, but I would not suggest that it would be easy or fast with my skill set.

Here's the hypothesis I would propose:

There is a text-only browser called Lynx from the University of Kansas. It's supposed to be scriptable. It would take some doing, but I imagine that one could get Lynx to go to the website and do a search within a specific page.

This browser cannot read iframes, javascripts, or much of anything besides the strait text on the web page. It's not pretty; designed for HTML 1.1; but, it is stable and useful.

Idea would be to write one script to control the browser. Get it to search the specific, pre-determined page (you yourself would have to determine the target, instead of the application doing lots of conditional searching). Print the contents of that page to a file in a specific location.

Meanwhile, another script would run a search on that file, periodically. This search would be for the target string, like a specific confirmation number you knew was coming down.

In this way, you could search a known target website for an expected string or condition. Idea would be to get the application to do 90% of the waiting for you, with the person minding it either responding to the program's alert, or checking up on the page themselves every so often.

Lynx's printing options include sending the printed content as an email.

I have never done any of this, but if I had to propose something, that would be it.

Notice, you would have to have just the right blend of conditions to make this plausible: experience with the browser, some elementary programming, a known set of possible answers, and the presentation of those answers in a convenient form of searchable text. That's a lot conditions.

Keep in mind, that by the time I got such a thing to work, whatever it was would have already happened.