CRAWL ERRORS - 404 not found.

[Jeet Kune Do]

I hope that somebody could help here. I have the following error in google webmaster tools:

http://mysite.com/content/mysite.com/content/1-delivery,404 (Not found),1 pages,
http://mysite.com/mycategorymysite.comi/mycategory/60-myproductname.html,404
http://mysite.com/mycategory2/mysite.com/mycategory2/34-myproductname.html,404
http://mysite.com/mycategory2/mysite.com/mycategory2/37-myproductname.html,404
http://mysite.com/mysite.com/1-categorymain,404
http://mysite.com/mysite.com/3-mycategory,404

The sources of error: (it's in the same order, please compare to the code up)

http://mysite.com/content/mysite.com/content/1-delivery,http://mysite.com/content/1-delivery,
http://mysite.com/mycategory/mysite.com/mycategory/60-myproductname.html,http://mysite.com/mycategory/60-myproductname.html,
http://mysite.com/mycategory2/mysite.com/mycategory2/34-myproductname.html,http://mysite.com/mycategory2/34-myproductname.html,
http://mysite.com/mycategory2/mysite.com/mycategory2/37-myproductname.html,http://mysite.com/mycategory2/37-myproductname.html,
http://mysite.com/mysite.com/1-categorymain,http://mysite.com/1-categorymain,
http://mysite.com/mysite.com/3-mycategory,http://mysite.comi/3-mycategory,

I replaced my real site name, category name and product. Because is an adult site. It's not for minors.
As you can see my domain name is repeted many times. The errors keep groing every few days when get indexed.

My prestashop version 1.3.2
Friendly URL: enabled
Cannonical URL: enabled (non www site)
In the google sitemap 1.6 there are not such urls. Everything looks normal

I would like to thank you for your time in advance!

Best regards!

[tomerg3]

What's the URL that is used by the Canonical URL module, I know it had some bugs in the past.

On a related note, I think Canonical is crap, since search engines still crawl the other duplicate pages, the best solution is to remove all instances of the duplicate URLs and use a 301 redirect for all the pages that search engines already crawled.

Check out http://www.prestashop.com/forums/viewthread/39736/

[Jeet Kune Do]

My settings for cannonical URL is:

Canonical URL
http(s):/                  //some/directories/a_prestashop_webpage.php

I insered mysite.com in the empty field in the module manager.

Is that correct? Should I turn it off?


In .htaccess I inserted the code below to redirect www.mysite.com to mysite.com. It could be the problem?

RewriteEngine on
RewriteCond %{HTTP_HOST} ^www.mysite.com$ [NC]
RewriteRule ^(.*)$ http://mysite.com/$1 [R=301,L]

I have more questions about your reddirect module.

I have indexed only the friendly URL in google, is your module still needed?

It works with such kind of modification?

http://www.openscriptsolution.com/2010/05/03/how-to-remove-category-id-from-url-of-prestashop-version-1-2-5/

Your module looks great and is also at affordable price. I wanna make sure those thinks before I buy it.



Thank you very much for your help. Sorry about my poor english.

Regards!

[tomerg3]

The module was not tested on modified URLs, I don't believe it will work without some modifications.

[Jeet Kune Do]

I turn canonical URL off, but I still get's errors and errors.

Please somebody help!

[Jeet Kune Do]

I hope that somebody could help here. I have the following error in google webmaster tools:

http://mysite.com/content/mysite.com/content/1-delivery,404 (Not found),1 pages,
http://mysite.com/mycategorymysite.comi/mycategory/60-myproductname.html,404
http://mysite.com/mycategory2/mysite.com/mycategory2/34-myproductname.html,404
http://mysite.com/mycategory2/mysite.com/mycategory2/37-myproductname.html,404
http://mysite.com/mysite.com/1-categorymain,404
http://mysite.com/mysite.com/3-mycategory,404

The sources of error: (it's in the same order, please compare to the code up)

http://mysite.com/content/mysite.com/content/1-delivery,http://mysite.com/content/1-delivery,
http://mysite.com/mycategory/mysite.com/mycategory/60-myproductname.html,http://mysite.com/mycategory/60-myproductname.html,
http://mysite.com/mycategory2/mysite.com/mycategory2/34-myproductname.html,http://mysite.com/mycategory2/34-myproductname.html,
http://mysite.com/mycategory2/mysite.com/mycategory2/37-myproductname.html,http://mysite.com/mycategory2/37-myproductname.html,
http://mysite.com/mysite.com/1-categorymain,http://mysite.com/1-categorymain,
http://mysite.com/mysite.com/3-mycategory,http://mysite.comi/3-mycategory,

As you can see my domain name is repeted many times. The errors keep groing every few days when get indexed.

My prestashop version 1.3.2
Friendly URL: enabled
Cannonical URL: enabled (non www site)
In the google sitemap 1.6 there are not such urls. Everything looks normal

I would like to thank you for your time in advance!

Best regards!

Can somebody help me, please? It's urgent!

Thank you in advance!

[tomerg3]

Is the canonical URL module enabled or disabled?

Do you see any of these URLs in the sitemap?

[Jeet Kune Do]

Cannonical URL is disabled.
Like I sayed in my first post in mine sitemap created by the module google sitemap 1.6 I don't have such's URL.

In google webmaster tools I have set preffered domain to non www.


Now I got 37 errors and It keep growing.

tomerg3, thanky you very much for your help!

[tomerg3]

It may take a long time before all those URLs are gone.
You can go over the site and make sure those URLs are not used anywhere when you view the source of the page.

[Jeet Kune Do]

Ok.

How can I redirected those link to the real location? For example:

http://mysite.com/mysite.com/contact-form.php

http://mysite.com/mysite.com/6-mycategory

I really appreciate your help!

[lynx887]

I have the same error in webmaster tools and now up to 355 errors.

http://www.mysite.nl/-gps/www.mysite.nl/-gps/33-navin-gps-backtracker.html
http://www.mysite.nl/-survival-kit/www.mysite.nl/-survival-kit/103-flint-striker-met-kompas.html
http://www.mysite.nl/-survival-kit/www.mysite.nl/-survival-kit/111-nato-waterbottle.html
http://www.mysite.nl/-survival-kit/www.mysite.nl/-survival-kit/121-silver-gripper.html

[Jeet Kune Do]

#tomerg3

Could you please reffer this topic also to other moderators or developer so we know if it's enought to disable Canonical URL module or it's some wird stuff going on.

My errors keep growing and my rank is penalized every day.


I really appresciate your help!


Best regards!

[tomerg3]

I have no way of contacting other moderators, and this is not really a moderator's job anyway.

You can post a bug report on this.

Hopefully it won't take google too long to stop crawling them.

[lynx887]

I disabled the Canonical URL module and removed the rewrite code below from the .htaccess file and this morning the crawl errors have reduced from 365 to 332, so things may be sorted now.

RewriteCond %{HTTP_HOST} ^mysite.nl$
RewriteRule ^/?$ "http\:\/\/www\.mysite\.nl" [R=301,L]

[Jeet Kune Do]

Thank you for sharring your experience.

I removed the code from .htaccess


The problem is that I need redirection from www to non www so I didn't did this before.

I hope that will fix the problem, but then we should try to find a solution for redirection.


I will update the post.


Thank you both for helping!

[eCasper]

I'm having exact the same problem. I've been trying to find a help on this since a month time now but still no luck!! I don't think removing redirection is the best solution for this because not only google webmaster has reported this problem and yahoo too.

I just redirected my thread also here. hope it will be easy for future searches.

[JTT]

I have the same problem...

In webmaster tools I can see 404 page not found on a strange URLs like :

http://www.mydomain.com/www.mydomain.com/something

Nobody knows ?

[lynx887]

Update, disabling Canonical URL and removing the rewrite code from the .htaccess has not worked, latest crawl by google now shows 445 errors. Issue must be somewhere else. No ideas?

[Jeet Kune Do]

For me it didn't work too yet!

Maybe need some time.


I will post update

P.S.- At least it' slow my errors coming up.

[eCasper]

I just removed the Canonical URL module and code for non www to www redirection from the .htaccess. Then I used the redirection which my hosting provider provides (Thanks Razaro for the idea). it added following piece of code to my .htaccess.

rewriteengine on
rewritecond %{HTTP_HOST} ^lankabiznet.com$
rewriterule ^$ "http\:\/\/www\.lankabiznet\.com\/" [R=301,L] #4ce92fbc42309

Good news is after three days of above modification my 404 errors reduced from 149 to 131. it reduce no of 404 errors every day it seems (yesterday it was 139).

[Jeet Kune Do]

#Duleepa Lakmal

Thank you for your reply.

Maybe the redirection that we find on prestashop forum doesn't fit for every hosting.

I will give it a try. I will contact my host providers and ask them for the redirection code.

[eCasper]

Bad news again!!! 404 errors suddenly have gone up to 249 today morning. looks like we yet to find a solution for this.

[Jeet Kune Do]

For me it also keep growing.

Damn! It's making me crazy.

[lynx887]

up to 906 now!

[Jeet Kune Do]

I post my .htaccess

Please somebody check if it's OK.

# .htaccess automaticaly generated by PrestaShop e-commerce open-source solution
# http://www.prestashop.com - http://www.prestashop.com/forums

# URL rewriting module activation
RewriteEngine on

# URL rewriting rules
RewriteRule ^([a-z0-9]+)\-([a-z0-9]+)(\-[_a-zA-Z0-9-]*)/([_a-zA-Z0-9-]*)\.jpg$ /img/p/$1-$2$3.jpg [QSA,L,E]
RewriteRule ^([0-9]+)\-([0-9]+)/([_a-zA-Z0-9-]*)\.jpg$ /img/p/$1-$2.jpg [QSA,L,E]
RewriteRule ^([0-9]+)(\-[_a-zA-Z0-9-]*)/([_a-zA-Z0-9-]*)\.jpg$ /img/c/$1$2.jpg [QSA,L,E]
RewriteRule ^lang-([a-z]{2})/([a-zA-Z0-9-]*)/([0-9]+)\-([a-zA-Z0-9-]*)\.html(.*)$ /product.php?id_product=$3&isolang;=$1$5 [QSA,L,E]
RewriteRule ^lang-([a-z]{2})/([0-9]+)\-([a-zA-Z0-9-]*)\.html(.*)$ /product.php?id_product=$2&isolang;=$1$4 [QSA,L,E]
RewriteRule ^lang-([a-z]{2})/([0-9]+)\-([a-zA-Z0-9-]*)(.*)$ /category.php?id_category=$2&isolang;=$1 [QSA,L,E]
RewriteRule ^([a-zA-Z0-9-]*)/([0-9]+)\-([a-zA-Z0-9-]*)\.html(.*)$ /product.php?id_product=$2$4 [QSA,L,E]
RewriteRule ^([0-9]+)\-([a-zA-Z0-9-]*)\.html(.*)$ /product.php?id_product=$1$3 [QSA,L,E]
RewriteRule ^([0-9]+)\-([a-zA-Z0-9-]*)(.*)$ /category.php?id_category=$1 [QSA,L,E]
RewriteRule ^content/([0-9]+)\-([a-zA-Z0-9-]*)(.*)$ /cms.php?id_cms=$1 [QSA,L,E]
RewriteRule ^([0-9]+)__([a-zA-Z0-9-]*)(.*)$ /supplier.php?id_supplier=$1$3 [QSA,L,E]
RewriteRule ^([0-9]+)_([a-zA-Z0-9-]*)(.*)$ /manufacturer.php?id_manufacturer=$1$3 [QSA,L,E]
RewriteRule ^lang-([a-z]{2})/(.*)$ /$2?isolang=$1 [QSA,L,E]

# Catch 404 errors
ErrorDocument 404 /404.php

[eCasper]

Hi Jeet Kune Do,
Do you keep your site in the maintenance mode for while and often?

[Jeet Kune Do]

#eCasper, thank you for your replay.

Not at all. But two week ago I have my site in maintenance for only two days.
In BO I have set my maintenance i.p. but the shop maintenance mode is disabled.

Should I delete my maintenance i.p?
Could you please share your experiences in details, you had also those kind of errors?


Best regards!

[dwilden]

Hi,

I have see the syntax of canonical meta tag. On some manuals the correct syntax is:

<link rel="canonical" href="http://www.mysite.com" />

My prestashop 1.3.2 add wrong meta tag, like this:

<link rel="canonical" href="www.mysite.com" />

I think this is a bug of canonical module, and you must add http:// or https:// in canonical configuration

Regards,
Dreamguard

[dwilden]

On Spanish forum, have the solution:

you must change on /modules/canonicalurl/canonicalurl.php all function hookheader with this,

public function hookHeader($params)
   {
       global $smarty, $content_base_url;

       $canonicalUrl = Configuration::get('CANONICAL_URL');
       $ps_request = str_replace(__PS_BASE_URI__, '', $_SERVER['REQUEST_URI']);
       if (strlen(Configuration::get('CANONICAL_URL')) > 0)
           $smarty->assign('canonical_url', $content_base_url.Tools::htmlentitiesUTF8(rawurldecode($_SERVER['REQUEST_URI'])));
       return $this->display(__FILE__, 'canonicalurl.tpl');
   }

[Jeet Kune Do]

I have disabled cannonical URL but nothing happened till I set the proper reddirection for my server from www to non www.
Now for a few days my error is going down every crawl.
I'm monitoring my errors and I was scared that the error would grow like for the other user in this threat.

Now I don't worry any more because I will try to apply your solution, so #Dreamguard

Thank you very much!!!!!!!!!!

I'm very glad that you find the solution for canonical URL so we could have this module ENABLED!

I will post if it will work for me.


THANK YOU FOR ALL THE CONTRIBUTORS OF THIS THREAT! I really appreciate.

Best regards!

[JTT]

Does this canonical url module work ok ?

I do not know if I have an old version, but the I had now crawl errors since I have disabled this module (and also the sorting of products).

Now when I enable all of it and make the change to the canonicalurl.php as mentioned above, I still get bad canonical url in the html header...

When I sort the product, I get the canonical URL with the parameter like "orderby" and so on. I think it is not good. It should be without that pointing only to the main category where the sorting was made....

[Jeet Kune Do]

1. I have disabled canonical URL
2. I have the following .htaccess (note: from www to non www redirection)

# .htaccess automaticaly generated by PrestaShop e-commerce open-source solution
# http://www.prestashop.com - http://www.prestashop.com/forums

# URL rewriting module activation
RewriteEngine on

# URL rewriting from www to non www
RewriteCond %{HTTP_HOST} ^www.mysite.com$
RewriteRule ^(.*)$ http://mysite.com [R=301] 

# URL rewriting rules
RewriteRule ^([a-z0-9]+)\-([a-z0-9]+)(\-[_a-zA-Z0-9-]*)/([_a-zA-Z0-9-]*)\.jpg$ /img/p/$1-$2$3.jpg [QSA,L,E]
RewriteRule ^([0-9]+)\-([0-9]+)/([_a-zA-Z0-9-]*)\.jpg$ /img/p/$1-$2.jpg [QSA,L,E]
RewriteRule ^([0-9]+)(\-[_a-zA-Z0-9-]*)/([_a-zA-Z0-9-]*)\.jpg$ /img/c/$1$2.jpg [QSA,L,E]
RewriteRule ^lang-([a-z]{2})/([a-zA-Z0-9-]*)/([0-9]+)\-([a-zA-Z0-9-]*)\.html(.*)$ /product.php?id_product=$3&isolang;=$1$5 [QSA,L,E]
RewriteRule ^lang-([a-z]{2})/([0-9]+)\-([a-zA-Z0-9-]*)\.html(.*)$ /product.php?id_product=$2&isolang;=$1$4 [QSA,L,E]
RewriteRule ^lang-([a-z]{2})/([0-9]+)\-([a-zA-Z0-9-]*)(.*)$ /category.php?id_category=$2&isolang;=$1 [QSA,L,E]
RewriteRule ^([a-zA-Z0-9-]*)/([0-9]+)\-([a-zA-Z0-9-]*)\.html(.*)$ /product.php?id_product=$2$4 [QSA,L,E]
RewriteRule ^([0-9]+)\-([a-zA-Z0-9-]*)\.html(.*)$ /product.php?id_product=$1$3 [QSA,L,E]
RewriteRule ^([0-9]+)\-([a-zA-Z0-9-]*)(.*)$ /category.php?id_category=$1 [QSA,L,E]
RewriteRule ^content/([0-9]+)\-([a-zA-Z0-9-]*)(.*)$ /cms.php?id_cms=$1 [QSA,L,E]
RewriteRule ^([0-9]+)__([a-zA-Z0-9-]*)(.*)$ /supplier.php?id_supplier=$1$3 [QSA,L,E]
RewriteRule ^([0-9]+)_([a-zA-Z0-9-]*)(.*)$ /manufacturer.php?id_manufacturer=$1$3 [QSA,L,E]
RewriteRule ^lang-([a-z]{2})/(.*)$ /$2?isolang=$1 [QSA,L,E]

# Catch 404 errors
ErrorDocument 404 /404.php

I confirm that my errors are going down every crawl.

I didn't try the modification of the cannonical URL. I think I wouldn't.
Finaly my errors are disapearing and I think I will take no risk.


Best regards!

[eCasper]

@ Jeet Kune Do I'm sorry I did not reply any sooner because I was keep monitoring the issue. yes I can confirmed that this is resolved.

@JTT after the modification allow some three weeks time for google to update their databases with your modification. regarding the order by issues did you check here?

[Jeet Kune Do]

#eCasper

No problem. Thank you for your confirmation!

[Jeet Kune Do]

How could I redirect 404 errors pages to permanent redirection 301?

For example:

FROM:

http://mysite.com/mycategory/mysite.com/mycategory/11-myproduct.html

TO:

http://mysite.com/mycategory/11-myproduct.html

I want to do this because some errors aren't dissapered.


Thank you in advance!

Regards!