Jump to content

robots.txt problem. Please advice!


x3n0m0rph
 Share

Recommended Posts

Hi,

 

I am having some strange problems with robots.txt. I went to webmaster tools and seen a warning - sitemap blocked by robots.txt. Went to sitemap section and the end of the sitemap looks like this:

# Files
Disallow: /ti-ai-uitat-parola
Disallow: /quick-order
Disallow: /guest-tracking
Disallow: /
Disallow: /
Disallow: /
Disallow: /
Disallow: /
Disallow: /
Disallow: /
Disallow: /
Disallow: /
Disallow: /
Disallow: /
Disallow: /

I also have html errors for some products that seemed to be restricted and 1312 warnings in sitemap section saying:

Url blocked by robots.txt - http://www.secretgarden.ro/1000-large_default/jerba-din-crizantema-alba-si-galbena.jpg

 

For the moment, I manualy edited robots.txt and deleted all Disallow: /

I tryed to regenerate robots.txt and sitemap - regenerated robots.txt is the same. I also restanted/ uninstall / replaced sitemap module - same result

 

Why is my robots.txt broken? How can I fix it? Please help!

Share this post


Link to post
Share on other sites

If you have a problem with the robots generation.

 

you have to check the function public function generateRobotsFile() you will find in mysite.com/controllers/admin/AdminMetaController.php

 

Overwrite the file from a clean PS folder.

 

also check if you have any override in mysite.com/override/controllers/admin/AdminMetaController.php

Edited by javsmile (see edit history)

Share this post


Link to post
Share on other sites

Done that, disabled overrides and nonnative modules, clear cache, regenerate robots.txt - same problem...

 

My site already droped from google first page on keywords....

 

I really need a solution...

 

For the moment, I disabled sitemap module (it seems to regenerate robots.thx with the sitemap) and manualy edited robots.txt ...

 

Please help...

Share this post


Link to post
Share on other sites

And.. I am back with the solution:

 

This was a bug related to languages and friendly url activation.

When you install prestashop you have multiple labguages. If you want to use only one language, like I do, you will deactivate other languages and delete them. This leaves some entries in yourdatabase -> ps_meta_lang -> there you will have entries with "NULL" in url_rewrite column. If you delete all this entries your robots.txt will be generated corectly next time.

 

PS: This forum becomes useless, since people are not interested in helping each other anymore.

  • Like 2

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...

Important Information

Cookies ensure the smooth running of our services. Using these, you accept the use of cookies. Learn More