update robots.txt

[rhandom]

How to update `robots.txt`? Seems Prestashop manages this itself

robots.txt was overwritten this morning and I see

# robots.txt automatically generated by PrestaShop e-commerce open-source solution
# http://www.prestashop.com - http://www.prestashop.com/forums
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html
User-agent: *
# Allow Directives
...

So how to add the following rule directly into the robots.txt file at the root of the site:

User-agent: OAI-SearchBot
Allow: /

This change should be safe, it doesn’t affect the security of the site, and it will significantly improve the site’s chances of appearing in new AI-driven search results.

Edited August 25, 2025 by rhandom (see edit history)

[rhandom]

Guess I can add after generation myself. Did add

# OpenAI Search Bot
User-agent: OAI-SearchBot
Allow: /

at the end of the file.

Only thing I worry about is that this is inside the current directory so Git overwrites will overwrite it. But I think I can merge from platform control panel 

Edited August 25, 2025 by rhandom (see edit history)

[martinvilla]

On 8/25/2025 at 5:23 PM, rhandom said:

How to update `robots.txt`? Seems Prestashop manages this itself

robots.txt was overwritten this morning and I see

# robots.txt automatically generated by PrestaShop e-commerce open-source solution
# http://www.prestashop.com - http://www.prestashop.com/forums
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html
User-agent: *
# Allow Directives
...

So how to add the following rule directly into the robots.txt file at the root of the site:

User-agent: OAI-SearchBot
Allow: /

This change should be safe, it doesn’t affect the security of the site, and it will significantly improve the site’s chances of appearing in new AI-driven search results.

I have the same question and I’m running into this exact issue with PrestaShop. Every time I try to modify robots.txt manually, it seems to get overwritten automatically. I also want to add a custom rule for a specific user agent, but it’s unclear where PrestaShop expects these changes to be made. From what I can tell, editing the file directly at the root isn’t reliable. I’m wondering if there’s a setting in the back office or a config file where this should be defined instead. If anyone knows the correct way to make this change persist, I’d really appreciate the guidance.

Edited 50 minutes ago by martinvilla (see edit history)

[Anahitrt]

PrestaShop regenerates robots.txt automatically so thereofre the manual edits at root will always get overwritten. 

The only option you got is to manage it at the back office under Shop Parameters then Traffic and SEO then SEO and URLs and regenerate robots.txt from there.

[PrestaHeroes.com]

here is a solution that might help

1. copy robots.txt to robots.custom.txt
2. edit robots.custom.txt and include custom directives etc.
3. add this to top of .htaccess file

RewriteEngine On

# Serve custom robots file instead of PrestaShop-generated one
RewriteRule ^robots\.txt$ /robots.custom.txt [L]