robots.txt excludes english language

[Sablab]

salve a tutti,

I've setup a prestashop v 1.7.6.1 which features two languages, italian and english. Both languages work well in general: any user can browser the frontend and switch among languages without problem (apart some random product details to be fixed). The only problem is that the robots.txt file generated via backend includes the following line at the end:

Files Disallow: /*en/

As a consequence, the whole english section of the site is excluded by the google crawler (i've verified that via google search console).

So my question is: can I manually remove that line? or do I need to set or amend some specific option so that that line won't be included again in case of a regeneration of the robots.txt file? or finally, is it correct this line to be there (in which case I would ask for the supposed reason, if any)?

Here follows the complete robots.txt, generated via shop parameters -> traffic & seo -> robots file generation (commenting the said line on the published website version)

# robots.txt automatically generated by PrestaShop e-commerce open-source solution
# http://www.prestashop.com - http://www.prestashop.com/forums
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html
User-agent: *
# Allow Directives
Allow: */modules/*.css
Allow: */modules/*.js
Allow: */modules/*.png
Allow: */modules/*.jpg
Allow: /js/jquery/*
# Private pages
Disallow: /*?order=
Disallow: /*?tag=
Disallow: /*?id_currency=
Disallow: /*?search_query=
Disallow: /*?back=
Disallow: /*?n=
Disallow: /*&order=
Disallow: /*&tag=
Disallow: /*&id_currency=
Disallow: /*&search_query=
Disallow: /*&back=
Disallow: /*&n=
Disallow: /*controller=addresses
Disallow: /*controller=address
Disallow: /*controller=authentication
Disallow: /*controller=cart
Disallow: /*controller=discount
Disallow: /*controller=footer
Disallow: /*controller=get-file
Disallow: /*controller=header
Disallow: /*controller=history
Disallow: /*controller=identity
Disallow: /*controller=images.inc
Disallow: /*controller=init
Disallow: /*controller=my-account
Disallow: /*controller=order
Disallow: /*controller=order-slip
Disallow: /*controller=order-detail
Disallow: /*controller=order-follow
Disallow: /*controller=order-return
Disallow: /*controller=order-confirmation
Disallow: /*controller=pagination
Disallow: /*controller=password
Disallow: /*controller=pdf-invoice
Disallow: /*controller=pdf-order-return
Disallow: /*controller=pdf-order-slip
Disallow: /*controller=product-sort
Disallow: /*controller=search
Disallow: /*controller=statistics
Disallow: /*controller=attachment
Disallow: /*controller=guest-tracking
# Directories for www.dolcimascolo.com
Disallow: /app/
Disallow: /cache/
Disallow: /classes/
Disallow: /config/
Disallow: /controllers/
Disallow: /download/
Disallow: /js/
Disallow: /localization/
Disallow: /log/
Disallow: /mails/
Disallow: /modules/
Disallow: /override/
Disallow: /pdf/
Disallow: /src/
Disallow: /tools/
Disallow: /translations/
Disallow: /upload/
Disallow: /var/
Disallow: /vendor/
Disallow: /webservice/
Disallow: /en/app/
Disallow: /en/cache/
Disallow: /en/classes/
Disallow: /en/config/
Disallow: /en/controllers/
Disallow: /en/download/
Disallow: /en/js/
Disallow: /en/localization/
Disallow: /en/log/
Disallow: /en/mails/
Disallow: /en/modules/
Disallow: /en/override/
Disallow: /en/pdf/
Disallow: /en/src/
Disallow: /en/tools/
Disallow: /en/translations/
Disallow: /en/upload/
Disallow: /en/var/
Disallow: /en/vendor/
Disallow: /en/webservice/
Disallow: /it/app/
Disallow: /it/cache/
Disallow: /it/classes/
Disallow: /it/config/
Disallow: /it/controllers/
Disallow: /it/download/
Disallow: /it/js/
Disallow: /it/localization/
Disallow: /it/log/
Disallow: /it/mails/
Disallow: /it/modules/
Disallow: /it/override/
Disallow: /it/pdf/
Disallow: /it/src/
Disallow: /it/tools/
Disallow: /it/translations/
Disallow: /it/upload/
Disallow: /it/var/
Disallow: /it/vendor/
Disallow: /it/webservice/
# Files
Disallow: /*en/
Disallow: /*it/password-dimenticata
# Sitemap
Sitemap: https://www.dolcimascolo.com/1_index_sitemap.xml

thanks everyone

side note: i've reposted this topic from the italian forum (https://www.prestashop.com/forums/topic/1015845-robotstxt-esclude-lingua-en/) since I've got no answer in 3 days: I hope it's not against forum rules..

[tomerg3]

It could be a PS bug, why not simply delete that line?

If you're worried about PS regenerating it, you can make the file read only.

[Sablab]

thanks for your answer, so you would confirm that behaviour might be a bug and not some option that needs to be set via backend?
If it's a bug, might be useful to report it somewhere?

thank you for your answer again

 

[tomerg3]

I am not 100% sure about that, perhaps the language was disabled when the robots file was generated.

It's also possible that something else got messed up with your specific site / installation.