No: blocked by robots.txt

[Luigi Donato]

Greetings,
I would like to make the tag links indexable, so I added the hash (#) to the rules concerned, then I added the same rules, but with Allow instead of Disallow.

The suspected lines are as follows:

Allow: /*?tag=
Allow: /*en/search?tag=
#Disallow: /*?tag=
#Disallow: /*en/search

Also I noticed there was an HTML tag:

<meta name="robots" content="{$page.meta.robots}">

So I created a rule to prevent it from appearing in the tag pages and I succeeded, but according to Google and also according to other sites, the pages are blocked by the robots.txt file..
Can anyone enlighten me?

[webprog]

There are two lines for tags can be in robots.txt:

Disallow: /*?tag=
Disallow: /*&tag=

So, you need to allow all of them.

[Luigi Donato]

Well, leaving aside the fact that I would only need those without &, I tried to add that rule to Allows too, but they still get blocked.
Other ideas?

[webprog]

Did you try to insert the tag page into google index manually, in google webmaster?

[Luigi Donato]

Just now, webprog said:

Did you try to insert the tag page into google index manually, in google webmaster?

Yes

[webprog]

Can you show, the browser code of your tag page?

[Luigi Donato]

Yes:

The only weird thing I notice is this part, in the var prestashop line:
 

{"title": "Search", "description": "", "keywords": "search", "robots": "noindex"}

[webprog]

Everything looks right.

I think better to try to change SearchController.php function in controllers folder:

public function getTemplateVarPage()
    {
        $page = parent::getTemplateVarPage();

        $page['meta']['robots'] = 'noindex'; 

        return $page;
    }

 

[Luigi Donato]

This would only have the effect of not making the meta robots appear, but that problem has already been fixed.
Anyway, to be safe I tried and nothing has changed.

[webprog]

I have tested on my hosting and got the same result as you.

But after I request Google to refresh the robots.txt - google have added to index the search page. And now it is ok.

So you need to request Google to refresh robots.txt - https://www.google.com/webmasters/tools/robots-testing-tool?siteUrl=

Google don't refresh robots.txt everytime when you add link.

[Luigi Donato]

On 7/17/2022 at 11:08 AM, webprog said:

I have tested on my hosting and got the same result as you.

But after I request Google to refresh the robots.txt - google have added to index the search page. And now it is ok.

So you need to request Google to refresh robots.txt - https://www.google.com/webmasters/tools/robots-testing-tool?siteUrl=

Google don't refresh robots.txt everytime when you add link.

I noticed it too yesterday and I solved it as you said, thanks anyway