Jump to content

Robot txt blocking too much!


Andy1

Recommended Posts

When I check Google Webmaster Tools it is telling me that 45 pages are restricted by the robots.txt file. When I look at which pages they are, it's product and manufacturer pages (one I want the search engines to crawl). The list of restricted pages is growing every day.

 

Here is my robots.txt file:

 

 

 

robots.txt automaticaly generated by PrestaShop e-commerce open-source solution

# http://www.prestashop.com - http://www.prestashop.com/forums

 

# This file is to prevent the crawling and indexing of certain parts

# of your site by web crawlers and spiders run by sites like Yahoo!

# and Google. By telling these "robots" where not to go on your site,

# you save bandwidth and server resources.

 

# For more information about the robots.txt standard, see:

# http://www.robotstxt.org/wc/robots.html

 

User-agent: *

# Directories

Disallow: /classes/

Disallow: /config/

Disallow: /download/

Disallow: /mails/

Disallow: /modules/

Disallow: /translations/

Disallow: /tools/

Disallow: /lang-en/

# Files

Disallow: /addresses.php

Disallow: /address.php

Disallow: /authentication.php

Disallow: /cart.php

Disallow: /discount.php

Disallow: /footer.php

Disallow: /get-file.php

Disallow: /header.php

Disallow: /history.php

Disallow: /identity.php

Disallow: /images.inc.php

Disallow: /init.php

Disallow: /my-account.php

Disallow: /order.php

Disallow: /order-opc.php

Disallow: /order-slip.php

Disallow: /order-detail.php

Disallow: /order-follow.php

Disallow: /order-return.php

Disallow: /order-confirmation.php

Disallow: /pagination.php

Disallow: /password.php

Disallow: /pdf-invoice.php

Disallow: /pdf-order-return.php

Disallow: /pdf-order-slip.php

Disallow: /product-sort.php

Disallow: /search.php

Disallow: /statistics.php

Disallow: /attachment.php

Disallow: /guest-tracking

Disallow: /*orderby=

Disallow: /*orderway=

Disallow: /*tag=

Disallow: /*id_currency=

Disallow: /*search_query=

Disallow: /*id_lang=

Disallow: /*back=

Disallow: /*utm_source=

Disallow: /*utm_medium=

Disallow: /*utm_campaign=

Disallow: /*n=

# Sitemap

 

Help, please I am getting very concerned about this.

 

Many Thanks

Andy

Link to comment
Share on other sites

Please, can anyone help?

I am concerned about the number of pages that are 'restricted by robot txt'. My robot.txt file isn't blocking product pages, yet Google Webmaster Tools is suggesting it is.

 

The crawl rate of Google has dropped significantly and none of the changes I made to the sites Meta data has been updated for a month, unlike other sites I have created.

 

I am thinking of putting my other prestahsop site on hold for now, as this seems to be a problem.

Link to comment
Share on other sites

When I check Google Webmaster Tools it is telling me that 45 pages are restricted by the robots.txt file. When I look at which pages they are, it's product and manufacturer pages (one I want the search engines to crawl). The list of restricted pages is growing every day.

 

Here is my robots.txt file:

 

 

 

 

 

Help, please I am getting very concerned about this.

 

Many Thanks

Andy

what ever you want google to crawl just delete from your robot.txt file.

http://www.robotstxt.org/orig.html

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...