Andy1 Posted September 14, 2011 Share Posted September 14, 2011 When I check Google Webmaster Tools it is telling me that 45 pages are restricted by the robots.txt file. When I look at which pages they are, it's product and manufacturer pages (one I want the search engines to crawl). The list of restricted pages is growing every day. Here is my robots.txt file: robots.txt automaticaly generated by PrestaShop e-commerce open-source solution # http://www.prestashop.com - http://www.prestashop.com/forums # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources. # For more information about the robots.txt standard, see: # http://www.robotstxt.org/wc/robots.html User-agent: * # Directories Disallow: /classes/ Disallow: /config/ Disallow: /download/ Disallow: /mails/ Disallow: /modules/ Disallow: /translations/ Disallow: /tools/ Disallow: /lang-en/ # Files Disallow: /addresses.php Disallow: /address.php Disallow: /authentication.php Disallow: /cart.php Disallow: /discount.php Disallow: /footer.php Disallow: /get-file.php Disallow: /header.php Disallow: /history.php Disallow: /identity.php Disallow: /images.inc.php Disallow: /init.php Disallow: /my-account.php Disallow: /order.php Disallow: /order-opc.php Disallow: /order-slip.php Disallow: /order-detail.php Disallow: /order-follow.php Disallow: /order-return.php Disallow: /order-confirmation.php Disallow: /pagination.php Disallow: /password.php Disallow: /pdf-invoice.php Disallow: /pdf-order-return.php Disallow: /pdf-order-slip.php Disallow: /product-sort.php Disallow: /search.php Disallow: /statistics.php Disallow: /attachment.php Disallow: /guest-tracking Disallow: /*orderby= Disallow: /*orderway= Disallow: /*tag= Disallow: /*id_currency= Disallow: /*search_query= Disallow: /*id_lang= Disallow: /*back= Disallow: /*utm_source= Disallow: /*utm_medium= Disallow: /*utm_campaign= Disallow: /*n= # Sitemap Help, please I am getting very concerned about this. Many Thanks Andy Link to comment Share on other sites More sharing options...
Andy1 Posted September 16, 2011 Author Share Posted September 16, 2011 Please, can anyone help? I am concerned about the number of pages that are 'restricted by robot txt'. My robot.txt file isn't blocking product pages, yet Google Webmaster Tools is suggesting it is. The crawl rate of Google has dropped significantly and none of the changes I made to the sites Meta data has been updated for a month, unlike other sites I have created. I am thinking of putting my other prestahsop site on hold for now, as this seems to be a problem. Link to comment Share on other sites More sharing options...
werepair Posted September 23, 2011 Share Posted September 23, 2011 When I check Google Webmaster Tools it is telling me that 45 pages are restricted by the robots.txt file. When I look at which pages they are, it's product and manufacturer pages (one I want the search engines to crawl). The list of restricted pages is growing every day. Here is my robots.txt file: Help, please I am getting very concerned about this. Many Thanks Andy what ever you want google to crawl just delete from your robot.txt file. http://www.robotstxt.org/orig.html Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now