...

View Full Version : does anyone know of a program to download source code of entire website



kevinkhan
10-19-2009, 12:15 PM
Im looking for a program that can download the source code of all the ads in this website http://www.donedeal.ie/find/all/all/Cork/ and then combine all the source files into one big file...

Does anyone know where i could get software for this?

mlseim
10-19-2009, 01:01 PM
Kevin,

It can't be done unless you have access to their database.
Even their RSS feed is only the 10 latest ads.

kevinkhan
10-19-2009, 02:10 PM
Really :(

Is there anyway of making a custom program that can do this kind of thing.

Phil Jackson
10-19-2009, 03:01 PM
I've wrote you some code to create an RSS feed of 200 of the newest items.. wait 5 mins

Phil Jackson
10-19-2009, 03:05 PM
I could run this every day from my end and you could use file_get_contents() to get the feed. OPutput is something like:


<rss version="2.0">
<channel>
<listing>
<image>http://photos.donedeal.ie/donedeal/photos2/SearchThumb?id=1912427</image>
<price>&euro;85</price>
<age>1&nbsp;min</age>
<text>Very Pretty Christening Gown , pictures do not do it justice, will fit 2-3 month old approx, it has a lot of detail with lace and pearl design.</text>
<title>Clothes</title>
</listing>
<listing>
<image>http://photos.donedeal.ie/donedeal/photos2/SearchThumb?id=1912423</image>
<price>&euro;5,250</price>
<age>2&nbsp;min</age>
<text>PRICE DROP

1991 Ford Transit Camper Van.
2 Berth.
Mileage 47,000 miles.

Feautures:
2 RING HOB
MICROWAVE OVEN.
CASSETTE TOILET/...</text>
<title>IMMACULATE FORD TRANSIT CAMPER VAN</title>
</listing>
<listing>
<image>http://photos.donedeal.ie/donedeal/photos2/SearchThumb?id=1903605</image>
<price>&euro;130</price>
<age>3&nbsp;min</age>
<text>4 Allessio 17" alloy wheels to suit Toyota, Honda, Opel &amp; Nissan. 4 studs on 100 dia. 2 with tyres (205/40/17). 1 is repaired and needs...</text>
<title>Alloy Wheels</title>
</listing>
<listing>
<image>http://photos.donedeal.ie/donedeal/photos2/SearchThumb?id=1912386</image>
<price>&euro;4,000</price>
<age>8&nbsp;min</age>
<text>Hunter 490 lifting keel mini cruiser for sale.

This excellent mini cruiser comes with mainsail &amp; racing main. 3 jibs, Spinnaker &amp; Spinnaker...</text>
<title>Trailer sailer for sale</title>
</listing>


P.S. never made a RSS feed before so dont know if its structured correctly.

Phil Jackson
10-19-2009, 03:32 PM
link to run script for testing purposes:

All zipped:
http://www.actwebdesigns.co.uk/codingforumhelp/RSS%20Feeder/RSS%20Feeder.rar

Main Code:



<?php

// http://www.plugnplaycms.co.uk
// http://www.actwebdesigns.co.uk
// luke@actwebdesigns.co.uk
// need some help? Let me know about it.

//SETUP
include("simplehtmldom/simple_html_dom.php");
$delayPerScan = array("1.5", "1.75", "1.95", "2", "2.5", "2.2", "2.7", "3", "3.1", "2.8");

$outputFile = "output/data.xml";
$pages = 10; //number of latest pages
//SETUP END

//MAIN
$data = "<rss version=\"2.0\">\n\t";
$data .= "<channel>\n\t\t";
for($x=0; $x!=$pages; $x++)
{
if($doneDealData = file_get_contents("http://www.donedeal.ie/find/all/all/Cork/?sort=publishDate+desc&start=".($x*15)))
{
sleep($delayPerScan[rand(0,9)]);
$html = new simple_html_dom();
$html = str_get_html($doneDealData);
$listings = $html->find("tr[title=For Sale]");
foreach($listings as $listing)
{
if($listing->find("img[alt=Photo]",0))
$Image = $listing->find("img[alt=Photo]",0)->src;
else
$Image = "Image Not Available";

if($listing->find("span[class=publishDate]",0))
$Age = $listing->find("span[class=publishDate]",0)->plaintext;
else
$Age = "No Age";

if($listing->find("span[class=price]",0))
$Price = $listing->find("span[class=price]",0)->plaintext;
else
$Price = "No Price Given";

if($listing->find("p[class=description]",0))
$Text = $listing->find("p[class=description]",0)->plaintext;
else
$Text = "No Description Given";

if($listing->find("span[class=header] a",0))
$Title = $listing->find("span[class=header] a",0)->plaintext;
else
$Title = "No Tilte Given";

$data .= "<listing>\n\t\t";
$data .= " <image>".trim($Image)."</image>\n\t\t";
$data .= " <price>".trim($Price)."</price>\n\t\t";
$data .= " <age>".trim($Age)."</age>\n\t\t";
$data .= " <text>".trim($Text)."</text>\n\t\t";
$data .= " <title>".trim($Title)."</title>\n\t\t";
$data .= "</listing>\n\t\t";
}
}
}
$data .= "</channel>\n\t";
$data .= "</rss>\n";

$fp = fopen($outputFile, 'w');
fwrite($fp, $data);
fclose($fp);

?>

kevinkhan
10-19-2009, 03:54 PM
Im afraid i cant get it working :(

Phil Jackson
10-19-2009, 03:56 PM
just nipping out will be half an hour

kevinkhan
10-19-2009, 04:09 PM
ok thanks...

tomws
10-19-2009, 04:10 PM
I guess it's only copyright violation when someone else does it.

From http://help.donedeal.ie/info/copyright-notice-legal-disclaimer


Copyright Notice

Material on this website is protected by copyright. Reproduction of material from this website is authorised for personal use. Reproduction for other uses is permitted, provided that the source is acknowledged. All material on this website, including, (but not limited to) design, text, graphics and all software compilations and underlying source code are the copyright of Done Deal Limited and/or DoneDeal.ie. ALL RIGHTS RESERVED.

mlseim
10-19-2009, 04:18 PM
Kevin,
Are you saying that this is your OWN website and you want to make one
file with all ads? Or is this someone else's site? I'm confused.

If you're trying to get all of the ads from someone else's site, you can't do it.
How would your script navigate all pages and parse the HTML?
You would need access to their database.

Everyone else is talking about making an RSS feed ... ???
From someone else's website?

kevinkhan
10-19-2009, 04:29 PM
No this is not my own website.. and i dont have access to the database...

Phil Jackson
10-19-2009, 04:42 PM
ah, well in that case, this was made to show how it COULD be done and not how you should do it. It would be a breach of copy write law and hold no responsibility for the use of what is posted above.



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum