Bug in robots.txt ?

[famousthimo]

Hi,

 

Prestashops build the robots.txt file telling google which directories not to index

 

# Directories

Disallow: */classes/

Disallow: */config/

Disallow: */download/

Disallow: */mails/

Disallow: */modules/

Disallow: */translations/

Disallow: */tools/

 

 

 

But it excludes more than just the main directories: 

 

Disallow: */modules/  

 

within this line it also excludes other directories like

 

www.mydomain.com/themes/default-bootstrap/css/modules/blockbestsellers/blockbestsellers.css 

www.mydomain.com/themes/default-bootstrap/css/modules/blockcart/blockcart.css

www.mydomain.com/themes/default-bootstrap/css/modules/blockcategories/blockcategories.css

www.mydomain.com/themes/default-bootstrap/css/modules/....

 

 

Google needs the css info 

[famousthimo]

Hi,

 

Prestashops build the robots.txt file telling google which directories not to index

 

# Directories

Disallow: */classes/

Disallow: */config/

Disallow: */download/

Disallow: */mails/

Disallow: */modules/

Disallow: */translations/

Disallow: */tools/

 

 

 

But it excludes more than just the main directories: 

 

Disallow: */modules/  

 

within this line it also excludes other directories like

 

www.mydomain.com/themes/default-bootstrap/css/modules/blockbestsellers/blockbestsellers.css 

www.mydomain.com/themes/default-bootstrap/css/modules/blockcart/blockcart.css

www.mydomain.com/themes/default-bootstrap/css/modules/blockcategories/blockcategories.css

www.mydomain.com/themes/default-bootstrap/css/modules/....

 

 

Google needs the css info 

 

This seems a real bug , because when i test my site with the mobile friendly test from google

 

It returns that my site isn't mobile friendly because google can't access the CSS info

 

When i leave the line Disallow: */modules/   out of my robots.txt , then google rates my site as MOBILE FRIENDLY

 

Anyone knows how to disallow the modules directory but how to ALLOW other directories who are also calles modules ??? like www.mydomain.com/themes/default-bootstrap/css/MODULES/blockbestsellers/blockbestsellers.css 

[Ben90]

Give a try

Allow: *modules/blockbestsellers

[famousthimo]

 

Give a try

Allow: *modules/blockbestsellers

Hi,

 

I had to add several allows for it to work.

 

I've added 

 

Allow: /*themes/default-bootstrap

Allow: /*modules/blockfacebook

Allow: /*modules/themeconfigurator

Allow: /*modules/paypal

 

Now it works fine. But if i ever update the robots.txt file through prestashop backoffice, it will be overwritten.

So hope Prestashop will fix this bug soon