Jump to content

Google master tool Warnings:Url blocked by robots.txt.


Recommended Posts

hi,People.

I really need some help. I received google master tool Warnings for some days.

 

Warnings

Url blocked by robots.txt.

Sitemap contains urls which are blocked by robots.txt.

1

 

Value:http://www.mysite.com/discount.php

-

 

following is rebots txt:

 

# robots.txt automaticaly generated by PrestaShop e-commerce open-source solution

# http://www.prestashop.com - http://www.prestashop.com/forums

 

# This file is to prevent the crawling and indexing of certain parts

# of your site by web crawlers and spiders run by sites like Yahoo!

# and Google. By telling these "robots" where not to go on your site,

# you save bandwidth and server resources.

 

# For more information about the robots.txt standard, see:

# http://www.robotstxt.org/wc/robots.html

 

User-agent: *

# Directories

Disallow: /classes/

Disallow: /config/

Disallow: /download/

Disallow: /mails/

Disallow: /modules/

Disallow: /translations/

Disallow: /tools/

Disallow: /lang-en/

# Files

Disallow: /addresses.php

Disallow: /address.php

Disallow: /authentication.php

Disallow: /cart.php

Disallow: /discount.php

Disallow: /footer.php

Disallow: /get-file.php

Disallow: /header.php

Disallow: /history.php

Disallow: /identity.php

Disallow: /images.inc.php

Disallow: /init.php

Disallow: /my-account.php

Disallow: /order.php

Disallow: /order-opc.php

Disallow: /order-slip.php

Disallow: /order-detail.php

Disallow: /order-follow.php

Disallow: /order-return.php

Disallow: /order-confirmation.php

Disallow: /pagination.php

Disallow: /password.php

Disallow: /pdf-invoice.php

Disallow: /pdf-order-return.php

Disallow: /pdf-order-slip.php

Disallow: /product-sort.php

Disallow: /search.php

Disallow: /statistics.php

Disallow: /attachment.php

Disallow: /guest-tracking

Disallow: /*orderby=

Disallow: /*orderway=

Disallow: /*tag=

Disallow: /*id_currency=

Disallow: /*search_query=

Disallow: /*id_lang=

Disallow: /*back=

Disallow: /*utm_source=

Disallow: /*utm_medium=

Disallow: /*utm_campaign=

Disallow: /*n=

# Sitemap

Sitemap: http://www.mysite.com/sitemap.xml

 

I can not find any wrong in this robot.txt. please help me to remove this

Warnings. thanks!thank!

Share this post


Link to post
Share on other sites
  • 3 months later...

hi,People.

I really need some help. I received google master tool Warnings for some days.

 

Warnings

Url blocked by robots.txt.

Sitemap contains urls which are blocked by robots.txt.

1

 

Value:http://www.mysite.com/discount.php

-

 

following is rebots txt:

 

# robots.txt automaticaly generated by PrestaShop e-commerce open-source solution

# http://www.prestashop.com - http://www.prestashop.com/forums

 

# This file is to prevent the crawling and indexing of certain parts

# of your site by web crawlers and spiders run by sites like Yahoo!

# and Google. By telling these "robots" where not to go on your site,

# you save bandwidth and server resources.

 

# For more information about the robots.txt standard, see:

# http://www.robotstxt.../wc/robots.html

 

User-agent: *

# Directories

Disallow: /classes/

Disallow: /config/

Disallow: /download/

Disallow: /mails/

Disallow: /modules/

Disallow: /translations/

Disallow: /tools/

Disallow: /lang-en/

# Files

Disallow: /addresses.php

Disallow: /address.php

Disallow: /authentication.php

Disallow: /cart.php

Disallow: /discount.php

Disallow: /footer.php

Disallow: /get-file.php

Disallow: /header.php

Disallow: /history.php

Disallow: /identity.php

Disallow: /images.inc.php

Disallow: /init.php

Disallow: /my-account.php

Disallow: /order.php

Disallow: /order-opc.php

Disallow: /order-slip.php

Disallow: /order-detail.php

Disallow: /order-follow.php

Disallow: /order-return.php

Disallow: /order-confirmation.php

Disallow: /pagination.php

Disallow: /password.php

Disallow: /pdf-invoice.php

Disallow: /pdf-order-return.php

Disallow: /pdf-order-slip.php

Disallow: /product-sort.php

Disallow: /search.php

Disallow: /statistics.php

Disallow: /attachment.php

Disallow: /guest-tracking

Disallow: /*orderby=

Disallow: /*orderway=

Disallow: /*tag=

Disallow: /*id_currency=

Disallow: /*search_query=

Disallow: /*id_lang=

Disallow: /*back=

Disallow: /*utm_source=

Disallow: /*utm_medium=

Disallow: /*utm_campaign=

Disallow: /*n=

# Sitemap

Sitemap: http://www.mysite.com/sitemap.xml

 

I can not find any wrong in this robot.txt. please help me to remove this

Warnings. thanks!thank!

 

Share this post


Link to post
Share on other sites

Hallo,

I have also the same problem and my prestashop it's the same 1.4.3. version.I read about this topic and I learned that somehow it's harmless for google crawls actually, but I can't explain details.I'm interested too in this subject if anybody could give some more details will be very appreciated.

 

Thanks

Share this post


Link to post
Share on other sites

Hello, newinprestashop, welcome to the Forums. We are happy to have you here. Take a look at this thread. http://www.prestashop.com/forums/topic/140883-robotstxt-and-webmaster-tools-error/

 

 

Many url are blocked by robots.txt and it's not bug, because many url aren't relevant to reference, and Google does not reference them.

Best regards,

Benjamin

Share this post


Link to post
Share on other sites
  • 2 years later...

Hello, newinprestashop, welcome to the Forums. We are happy to have you here. Take a look at this thread. http://www.prestashop.com/forums/topic/140883-robotstxt-and-webmaster-tools-error/

 

 

Many url are blocked by robots.txt and it's not bug, because many url aren't relevant to reference, and Google does not reference them.

Best regards,

Benjamin

 

This does not hold true for with my issue..

 

My shop has thousands of products and out of the entire site, I'm receiving Sitemap block error for only 2 of my products. Those 2 products are blocked by robots.txt and I can't figure out why or how to resolve.. I'm thinking it may be a bug.

Share this post


Link to post
Share on other sites
  • 5 years later...
On 2/18/2015 at 6:49 PM, thx2012 said:

 

This does not hold true for with my issue..

 

My shop has thousands of products and out of the entire site, I'm receiving Sitemap block error for only 2 of my products. Those 2 products are blocked by robots.txt and I can't figure out why or how to resolve.. I'm thinking it may be a bug.

Same here. Old post, after 5 years, but did you find solution?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...

Important Information

Cookies ensure the smooth running of our services. Using these, you accept the use of cookies. Learn More