Jump to content

Problem robot.txt and index


Recommended Posts

Hello guys,

We have problem with our robot.txt file and google index.

Problem 1)

sitemap: https://www.wdracing.eu/it/module/lgsitemaps/sitemap?fc=module&name=sitemap_1

Google says that some pages are blocked by robot.txt file

for example link: https://www.wdracing.eu/it/pastiglie-e-pinze-freno/25646-distanziale-braking-honda-crf-19-270mm.html

our robot.txt file: https://www.wdracing.eu/robots.txt

When using googlebot tool, it doen not find errors

what can we do ?

problem 2)

some pages are not valid.  we have:

23691 valid pages

4200 pages with errors

8024 exluded pages

problem 3)

If from search console we go in --> products--> missing field description, brand and others,  i see 0 valid pages e 20000 valid with warnings. 

we have a lot of link of category and not products. see image below

here i have found a similar problem but is not clear how to fix it: https://support.google.com/webmasters/thread/7159245?hl=en

problem 4)

if i analyze a product link i see a lot of products and not only the one i have analyzed. see below link

https://search.google.com/structured-data/testing-tool#url=https%3A%2F%2Fwww.wdracing.eu%2Fit%2Fcavalletti-trasporto-moto%2F670-260435-cavalletto-alzamoto-cross-enduro-racetech.html%23%2F7120-tipo-motocross%2F30185-aa_colore-giallo

problem 5 )

really few of the product images are on google images

Please can you help me ?

Many thanks to comunity

Schermata 2020-04-26 alle 11.36.48.png

Share this post


Link to post
Share on other sites
On 4/26/2020 at 4:23 PM, Ste-WD said:

Hello guys,

We have problem with our robot.txt file and google index.

Problem 1)

sitemap: https://www.wdracing.eu/it/module/lgsitemaps/sitemap?fc=module&name=sitemap_1

Google says that some pages are blocked by robot.txt file

for example link: https://www.wdracing.eu/it/pastiglie-e-pinze-freno/25646-distanziale-braking-honda-crf-19-270mm.html

our robot.txt file: https://www.wdracing.eu/robots.txt

When using googlebot tool, it doen not find errors

what can we do ?

problem 2)

some pages are not valid.  we have:

23691 valid pages

4200 pages with errors

8024 exluded pages

problem 3)

If from search console we go in --> products--> missing field description, brand and others,  i see 0 valid pages e 20000 valid with warnings. 

we have a lot of link of category and not products. see image below

here i have found a similar problem but is not clear how to fix it: https://support.google.com/webmasters/thread/7159245?hl=en

problem 4)

if i analyze a product link i see a lot of products and not only the one i have analyzed. see below link

https://search.google.com/structured-data/testing-tool#url=https%3A%2F%2Fwww.wdracing.eu%2Fit%2Fcavalletti-trasporto-moto%2F670-260435-cavalletto-alzamoto-cross-enduro-racetech.html%23%2F7120-tipo-motocross%2F30185-aa_colore-giallo

problem 5 )

really few of the product images are on google images

Please can you help me ?

Many thanks to comunity

Schermata 2020-04-26 alle 11.36.48.png

Hello there,

Hope you are fine.

Problem 1)

sitemap: https://www.wdracing.eu/it/module/lgsitemaps/sitemap?fc=module&name=sitemap_1

Google says that some pages are blocked by robot.txt file

for example link: https://www.wdracing.eu/it/pastiglie-e-pinze-freno/25646-distanziale-braking-honda-crf-19-270mm.html

our robot.txt file: https://www.wdracing.eu/robots.txt

When using googlebot tool, it doen not find errors

what can we do ?

Reply: The URL you have given in example is for search query of your website and also it is blocked by Robots.txt as we have checked, but this is not a problem it should be blocked.

problem 2)

some pages are not valid.  we have:

23691 valid pages

4200 pages with errors

8024 exluded pages

Reply: What type of pages they are? Can you show me here.

problem 3)

If from search console we go in --> products--> missing field description, brand and others,  i see 0 valid pages e 20000 valid with warnings. 

we have a lot of link of category and not products. see image below

here i have found a similar problem but is not clear how to fix it: https://support.google.com/webmasters/thread/7159245?hl=en

Reply: There are some missing fields in your product schema why search console shows a warning. You need to contact your developer regarding this.

problem 4)

if i analyze a product link i see a lot of products and not only the one i have analyzed. see below link

https://search.google.com/structured-data/testing-tool#url=https%3A%2F%2Fwww.wdracing.eu%2Fit%2Fcavalletti-trasporto-moto%2F670-260435-cavalletto-alzamoto-cross-enduro-racetech.html%23%2F7120-tipo-motocross%2F30185-aa_colore-giallo

Reply: Same as reply 4

problem 5 )

really few of the product images are on google images

Please can you help me ?

Reply: Need to check product schema whether property is set for images or not.

Thanks!!!

Share this post


Link to post
Share on other sites
4 hours ago, CedCommerce Team said:

Hello there,

Hope you are fine.

Problem 1)

sitemap: https://www.wdracing.eu/it/module/lgsitemaps/sitemap?fc=module&name=sitemap_1

Google says that some pages are blocked by robot.txt file

for example link: https://www.wdracing.eu/it/pastiglie-e-pinze-freno/25646-distanziale-braking-honda-crf-19-270mm.html

our robot.txt file: https://www.wdracing.eu/robots.txt

When using googlebot tool, it doen not find errors

what can we do ?

Reply: The URL you have given in example is for search query of your website and also it is blocked by Robots.txt as we have checked, but this is not a problem it should be blocked.

REPLY 2 : you are right. but what if i want that products to be index ? beacause we have made a lot of redirects to that products from old website. beacause they are "general" product that send to a specific product when you select your bike. i can consider to allow the search query in robot.txt ?

problem 2)

some pages are not valid.  we have:

23691 valid pages

4200 pages with errors

8024 exluded pages

Reply: What type of pages they are? Can you show me here.

REPLY 2: yes see below

image.png.ad19fe144387eed0669f094df72fe462.png

problem 3)

If from search console we go in --> products--> missing field description, brand and others,  i see 0 valid pages e 20000 valid with warnings. 

we have a lot of link of category and not products. see image below

here i have found a similar problem but is not clear how to fix it: https://support.google.com/webmasters/thread/7159245?hl=en

Reply: There are some missing fields in your product schema why search console shows a warning. You need to contact your developer regarding this.

REPLY 2: i have read that one of these modules can solve the product schema problem: 

https://addons.prestashop.com/en/seo-natural-search-engine-optimization/44883-clean-and-optimize-microdata-and-open-graph-seo.html

https://addons.prestashop.com/en/seo-natural-search-engine-optimization/42397-full-json-ld-microdata-and-open-graph-seo.html

https://addons.prestashop.com/en/seo-natural-search-engine-optimization/39955-seo-schema-markup-structured-data-rich-snippet.html

https://addons.prestashop.com/en/url-redirects/19643-advanced-seo-friendly-urls.html#specifications

 

4 hours ago, CedCommerce Team said:

do you agree ? what do you suggest ?

problem 4)

if i analyze a product link i see a lot of products and not only the one i have analyzed. see below link

https://search.google.com/structured-data/testing-tool#url=https%3A%2F%2Fwww.wdracing.eu%2Fit%2Fcavalletti-trasporto-moto%2F670-260435-cavalletto-alzamoto-cross-enduro-racetech.html%23%2F7120-tipo-motocross%2F30185-aa_colore-giallo

Reply: Same as reply 4

problem 5 )

really few of the product images are on google images

Please can you help me ?

Reply: Need to check product schema whether property is set for images or not.

REPLY 2 I think that they are related to previus errors. if we solve them the images will be index

Thanks!!!

Hello, please see my answers REPLY 2

Edited by Ste-WD (see edit history)

Share this post


Link to post
Share on other sites
  • 9 months later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...

Important Information

Cookies ensure the smooth running of our services. Using these, you accept the use of cookies. Learn More