Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 7 of 7
  1. #1
    Regular Coder Common's Avatar
    Join Date
    Jan 2009
    Location
    Glasgow, UK
    Posts
    132
    Thanks
    16
    Thanked 13 Times in 13 Posts

    Google Webmaster Tools - Robots.txt Fetch Problem

    Hello there

    Hopefully someone here might have some suggestions, as I can't get anything useful from Google Product Support!

    Basically, I launched my website about 7 days ago (link in sig). Before that it was in "maintenance mode", therefore no-one (including bots) could access the site or any of its files, including robots.txt.

    Naturally, this meant that google webmaster tools was reporting that it could not crawl, which was fine. The website has now been live for 7 days, and I can access robots.txt no problem, however google webmaster tools still claims to be unable to do so. Even when I successfully "Fetch As Google" and index all linked pages, the dashboard refuses to show the green tick beneat robots.txt fetch.

    Has anyone else had problems of this nature? Any idea how I can resolve it?

    Many thanks.

  • #2
    New to the CF scene
    Join Date
    Mar 2013
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Have you opened your robot.txt file? It should read something like this;

    User-Agent: *
    Allow: /

    and not;

    User-agent: *
    Disallow: /

    Also, you might want to double check the meta-tags in your HTML files and make sure this is NOT included:

    <meta name="robots" content="noindex, nofollow">

  • #3
    Regular Coder Common's Avatar
    Join Date
    Jan 2009
    Location
    Glasgow, UK
    Posts
    132
    Thanks
    16
    Thanked 13 Times in 13 Posts
    My robots.txt file can be seen here:

    http://www.getoutglasgow.com/robots.txt

    As far as I'm aware, this is a valid format. I do not use robots meta tags on the pages, as I am happy for everything to be followed and indexed.

  • #4
    Regular Coder patryk's Avatar
    Join Date
    Oct 2012
    Location
    /dev/couch
    Posts
    398
    Thanks
    2
    Thanked 64 Times in 64 Posts
    it sometimes takes up to a month for google to start indexing new site.
    u just have to wait a bit longer
    plus in webmaster tools informations are not displayed in real time for some reason.

  • Users who have thanked patryk for this post:

    Common (03-22-2013)

  • #5
    Regular Coder
    Join Date
    Mar 2013
    Posts
    262
    Thanks
    4
    Thanked 67 Times in 67 Posts
    Quote Originally Posted by Common View Post
    My robots.txt file can be seen here:

    http://www.getoutglasgow.com/robots.txt

    As far as I'm aware, this is a valid format. I do not use robots meta tags on the pages, as I am happy for everything to be followed and indexed.
    Your site seems to be getting indexed just fine on my end. Search engines treat sites with www and those without as two separate sites.

    Instead of typing this into Google...

    Code:
    site:www.getoutglasgow.com
    Try this..

    Code:
    site:getoutglasgow.com
    156 pages are indexed on my end.

  • #6
    New to the CF scene
    Join Date
    Feb 2013
    Location
    Hamilton
    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Some users would to like for making the use of the robot.txt file because it is one type of restriction on your site which can give instruction to the search engine crawler which pages should be crawled.

  • #7
    New Coder
    Join Date
    Jan 2013
    Posts
    71
    Thanks
    3
    Thanked 2 Times in 2 Posts
    Goolge used to crawl the pages in 2 or 3 days minimum and maximum up to 1 week after you have given the fetch in web master tools.If it not crawled till the maximum period it seems problem in updating any of the coding in your pages.Check again .If you given the coding correctly.then try to fetch one more time.


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •