Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 7 of 7
  1. #1
    New to the CF scene
    Join Date
    May 2014
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Need to Download Domain

    I am not familiar with much coding, but I am looking for a way to download an entire website domain and all the children as html files in a directory on my computer. For example, I use the code with a specified website, it starts a download process, once it is done, I look in my folder and there is the entire website in the form of html files.

    I had previously been using the wget command (I am on a mac) and it is so helpful. However, I cannot stand trying to get every single html file with that one by one for a domain. I primarily want to download some wikis. If somebody can help me get a code that can do this that would be very helpful.

  • #2
    Master Coder felgall's Avatar
    Join Date
    Sep 2005
    Location
    Sydney, Australia
    Posts
    6,639
    Thanks
    0
    Thanked 649 Times in 639 Posts
    If it is your domain where you have login details then you can use any FTP program to make a copy of the entire domain.
    Stephen
    Learn Modern JavaScript - http://javascriptexample.net/
    Helping others to solve their computer problem at http://www.felgall.com/

    Don't forget to start your JavaScript code with "use strict"; which makes it easier to find errors in your code.

  • #3
    New to the CF scene
    Join Date
    May 2014
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    No it is not my domain, but I am sure there must be an easier way to get all of the child pages of a domain than using wget on every single individual one.

  • #4
    New to the CF scene
    Join Date
    May 2014
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by milocricket View Post
    No it is not my domain, but I am sure there must be an easier way to get all of the child pages of a domain than using wget on every single individual one.


    I got it working for the most part. This is what I got.

    Code:
    wget http://example.com -r --limit-rate=60k -l 1 --convert-links --html-extension --recursive -P/tmp/<name of folder> -w 1 -t 1 -T 5 -nd -k -e "robots=off"
    But I have good news and bad news. Good news is that I can get all the pages into one folder, the bad news is I need it in a hierarchy. I did it right at some point earlier, but I scrapped that code. Now I need it back.
    Last edited by VIPStephan; 05-17-2014 at 02:17 PM. Reason: use example.com as placeholder domain, we don’t wanna promote third-party webhosts

  • #5
    New to the CF scene
    Join Date
    May 2014
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    2 things, I cannot find and edit button and the example url I put in turned into some domain link thing.

  • #6
    Master Coder felgall's Avatar
    Join Date
    Sep 2005
    Location
    Sydney, Australia
    Posts
    6,639
    Thanks
    0
    Thanked 649 Times in 639 Posts
    Quote Originally Posted by milocricket View Post
    No it is not my domain, but I am sure there must be an easier way to get all of the child pages of a domain than using wget on every single individual one.
    Do you have the owner's permission to copy all their content or are you stealing it?

    If you have permission then perhaps the owner can provide you with an easy way to get the copy.
    Stephen
    Learn Modern JavaScript - http://javascriptexample.net/
    Helping others to solve their computer problem at http://www.felgall.com/

    Don't forget to start your JavaScript code with "use strict"; which makes it easier to find errors in your code.

  • #7
    New to the CF scene
    Join Date
    Apr 2014
    Location
    Pakistan
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I use IDM for my it have been option to that download all of coding from the website. Go to IDM in the top task bar is option click in Grabber and put your required url in this portion. Click on download button and save all file into your computer.


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •