Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 7 of 7
  1. #1
    New Coder
    Join Date
    Mar 2004
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts

    preloading a website server sided..

    The site I am developing loades the search results of several other search engine (specifically travel engines) I post all results using IFRAMES embedded in DIVs. That means I load all results at one and a menu allows the user to select the layer (with particular vendor's results) he wants to see. If someone has a slower connection, the request sometimes times out on the vendor's site and the user would have to reload the frame in order to get the results. What I am thinking is that maybe if I preload all search results serversided, maybe I could then somehow pass the results on to the user, once they finish downloading on the server side. The user would only be waiting for my server to show the results and not for the vendor's server.

    Take a look, www.travelgrove.com just use the air search, it's not completely done but mostly works.

    Thanks for all your support, if you can point me into the right direction that would be great.

    Peter

  • #2
    raf
    raf is offline
    Master Coder
    Join Date
    Jul 2002
    Posts
    6,589
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Euh. Maybe i'm a bit comfused or i completely don't undestand it or so, but why do you assue that it would make a difference if the data issent from your server instead of from the vendors server?
    Isn't the problem that you send all the data at once to allow client side navigation then, without any server interaction?

    I mean, this sounds like the typical 'Do i preload and have a long initial downloadtime but fast navigation afterwards' versus the 'I only send the minimum required data and process each individual datarequest seperate at the server' dilemma.

    Maybe i just don't understand, but it seems to me that you will need to break down the data and send only the data that is requested. Storing the data for each individual session on the server and then look it up there before sending it to the client (i suppose that is what you mean with proloading it serverside??) is probably not a realy efficinent or even workable option because it will only offer realy limited flexability and you will never be able to support this setup on a high load website.

    The only logical approaches would be
    - to periodially load the venders complete db (or the relevant parts) and then transform them all into the same format (same variables, same valueranges) and store them in a db on your server. From there on, it's just a regular db-driven search and filter app. But i don't know if this is possible in your case
    - to first refine the search on the client, before launches it to the 3party searchengines --> for instance, let the client first select the vendor before lounching the search.

    But you seem to think you could easely set up some sort of meta-spider that will store and preproces the different searchengines results, and then spoonfeed it to the client or let the client filter on these server side stored and proprocessed resultsets or so. The first requirement would then be the format of the returned searchresult. This should be an XML file or so. Preferable ones that all use the same schema.
    If that requirement isn't fulfilled, then I frankly doubt you will be able to pull this off.

    Maybe you need to create two versions (broadband and 56k) ?
    Posting guidelines I use to see if I will spend time to answer your question : http://www.catb.org/~esr/faqs/smart-questions.html

  • #3
    New Coder
    Join Date
    Mar 2004
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Euh indeed.. :)

    First of all thank you very much for your detailed answer. I am looking for some hints to just see whether the things I want to do are possible or not, once I believe that they are, I dig in and I am sure I can find a good solution. So let me explain a few thing one more time, which should help you understand a bit better.

    There are several travel sites out there, that do a basic comparison of deals (Tripadvisor used to do this, Bookingbuddy does it and a few others). Some allow you to preselect up to 3 engines and open pop-ups to display the each result page (I think that solution stinks! It's not very userfriendly and a terrible marketing solution since merchants are automatically charged CPC usually). My search part is supposed to help out both advertisers as well as customers, I want more transparency and easier searches for consumers, and better trackability for marketers.

    I am not sure if you have visited my site, but basically right now it is set up the following way:
    I generate a dynamic URL using the parameters callected with my search mask. I pull all result pages in using IFRAMEs and embed those in DIVs. This way people can click between results without reloading at all. Naturally the download times are quite heavy. What's worse, I cannot do a body onload approach to preload the entire page, because some search engines regularly time out (this happens with almost every travel site once in a while). Whenever they time out, you would have to reload the frame to get the results (not nice)...
    When using an onload function, somtimes it will never be executed, because the body is not considered to be fully loaded. So what I do is I run a timer instead, which in term gives 15 secs to load data. Afterwards, some results are there, others still show the loading page of the particular travel site (I don't mind that, since most of them open in about 20 secs).

    Here is what I was thinking: Since only slow connections seem to time out with the travel merchants' servers, I could download the data first to my server, which shouldn't take too long. Then once a site has been retrieved I would pass it on to the user. As I download the second piece I pass that on again and so forth. I can't do the entire site at once, cause some sites like Orbitz can take a minute of loading before showing a result, and that has nothing to do with slow downloads. So if I wanted to download an entire page first, I would have to wait for all to load.

    Anyways, I agree that your 2 solutions might be my best odds, possibly ask the user which 3 engines he wants to load first, then while viewing already load the next 3 on to the server, and simply put a next 3 results button, then the user would download those from my server and so on.
    Your second solution to do 2 versions also makes sense, I have thought about this for a long time, and figured that it will not be too long until things switch to DSL completely, so maybe I will leave the broadband version for now.

    Last but not least, unfortunately I cannot spider the results, I can also not download all search results, there are too many options and too many vendors. You would be running tons of servers 24/7 and still be outdated. XML feeds are also not available yet in the travel industry, with exception to some hotel and cruise vendors, but that's it. SO I am stuck...

    Let me know if you have any other ideas. Again, you can check out the page to get a better understanding www.travelgrove.com

    Thanks a lot again...

  • #4
    raf
    raf is offline
    Master Coder
    Join Date
    Jul 2002
    Posts
    6,589
    Thanks
    0
    Thanked 0 Times in 0 Posts
    No, i didn't visit your site. I don't need to to imagen what i going on, what takes up the loadingtime and what possible sollutions are : you are loading the results of 10 sites at once.

    It would be a lott easier and would just look the same and offer the same fuctionality. If you just had 10 links to your vendors, with the required querystring values to initiate a search on that vendors site.
    When the user then hits one of these links, he will indeed need to wait a few seconds but he'll see the same result and i doubt he would mind that.

    But your problem/dilemma is not something special or unique or so.
    It's like in any other db-driven application. If i had a table with 500 products, then i could also load the complete table and use some clientside script to then browse through it.
    Once the 500 records are loaded, it would be blazing fast, but the initial load would be realy long + inefficient since most of the records wount be used.
    A far more efficient way is then to load the first 15 and then do some recordset paging (only selecting the next block of 15 records).--> much faster initial load, less datatrafic but slightly higher responsetimes to navigate through the complete set

    You have the same situation, only here, it aren't blocks of records, but the resultpages of a searchengine. But you could just as well only load the top bar with your 10 buttons, and then load the requested page in the bottom part of your screen. So just only one page at a time, instead of all the 10 pages and then navigate through the set clientsided.
    What you do now is just the same as if i would load my 500 records in sets of 50 that are all stored in a div, with then 10 links o these divs.
    Nobody would do that, so why should you take on such an approach for your site? Because the client then needs to wait a few seconds after hitting one of the buttons ?

    The only situation where you would indeed need to initiate the 10 searches all at once before displaying content, is when you would be spidering the returned results, or if you would do some additional comparisons or some kind of other manipulation. You know, some cross site processing or taking out the content of the returned pages and wrap your own layout around it. But as far as i can see, you launch the 10 searches and then display the 10 resulting pages. So your site is more some sort of portal with a 'type once, search on 10 sites' feature. A usufell and clientfriendly feature, but i don't see any reason to start preloading the searchresults from these 10 sites, simply because it doesn't add any extra functionality + because of the exact reason why you post here: it has a to high initial load time.

    Just my opinion.
    Posting guidelines I use to see if I will spend time to answer your question : http://www.catb.org/~esr/faqs/smart-questions.html

  • #5
    New Coder
    Join Date
    Mar 2004
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts

    the reason why I am doing this is simple

    travel searches can take forever, usually each vendor about 20 secs or so some will take a minute. The data put through at the same time is about 30-40k per page, so not very big. to download 400k may take 1 minute or so, but downloading at request will take 10x 20secs + which is much slower. You see, the amount of data loaded is not proportional with the time it takes from request to download. This would be different for an image gallery.

    Well, we will see how I solve this, I am very much leaning towards showing 3 - 5 results at once and then having a "next button". I would much rather see a better solution, but oh well.

    Is there a way to turn off pictures inside an iframe on the client side> that would make things much much faster, and would be an easy way to offer a 56k modem. dunno how this would look but it would be worth a shot.

    Thanks for your input.

    Peter

  • #6
    raf
    raf is offline
    Master Coder
    Join Date
    Jul 2002
    Posts
    6,589
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by suhayda
    travel searches can take forever, usually each vendor about 20 secs or so some will take a minute. The data put through at the same time is about 30-40k per page, so not very big. to download 400k may take 1 minute or so, but downloading at request will take 10x 20secs + which is much slower. You see, the amount of data loaded is not proportional with the time it takes from request to download. This would be different for an image gallery.
    Hmm. There is not much point in arguing about it. I would be very simple to set up the 'search on request' framework and check if the responsetimes are indeed 20 seconds. (Not on my PII with an ADSL connection anyway.
    Quote Originally Posted by suhayda
    Well, we will see how I solve this, I am very much leaning towards showing 3 - 5 results at once and then having a "next button". I would much rather see a better solution, but oh well.
    Good luck
    Quote Originally Posted by suhayda
    Is there a way to turn off pictures inside an iframe on the client side> that would make things much much faster, and would be an easy way to offer a 56k modem. dunno how this would look but it would be worth a shot.
    I don't think you can and most pages would stop woring because images are used for functional parts in these sites (tsj tsj tsj). Anyway, it would be not done since it's the client that should be able to decide about this. You can turn of loading images in your browser, but i hardly ever seen a 56k user do that, so i don't think most users want it ...
    It's also not done to trap oher sites inside a frame ...
    It's a shame not all pages are sent as XML which would allow us to make more integrated and faster composed and loaded pages, but ... there's still a long way to go before real data brokering will become reality...
    Posting guidelines I use to see if I will spend time to answer your question : http://www.catb.org/~esr/faqs/smart-questions.html

  • #7
    New Coder
    Join Date
    Mar 2004
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts

    thanks

    it's been great hearing your opinons. I appreciate it very much. I might post here again, if I have come up with a good solution.
    Thanks and best regards,

    Peter


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •