84 replies to this topic

[tomerg3]

There is a big problem with extra variables on pages which list products (category, search, best seller, specials and new products).

The sort by, # or results per page + language flags create new links with a potentially endless number of variables.

Example:

/best-sales.php?isolang=hi&n=15&p=3&id_lang=23&orderby=price&orderway=desc
/5-prestashop-modules?n=10&isolang=id&;id_lang=13&orderby=name&orderway=asc&id_lang=13&id_category=5
/5-prestashop-modules?n=10&isolang=id&id_lang=13&orderby=name&orderway=asc

While those URLs are not a problem for regular users, search engines that crawl your site find those links, then they crawl those links and find more new links on them.

For search engines, the sort order of products on a page, the number of result per page, and a language change variable are not needed, in fact, they just add new links for them to crawl with virtually the same data.

I estimate that the extra URLs that search engines crawl are about 50-100+ times more than that actual URLs that are valid and need crawling, this results in a lot of meta tag duplication, very similar content, and creates a huge load on your site when they crawl pages that are not really needed.

In the past I have attempted to address this by giving the page unique meta tags (see http://www.prestasho...ewthread/52665/), however, that was not a complete solution for two reasons:

1) Search engines still see and crawl those pages, and their content is virtually the same.

2) When the order of the extra variables would be different, the meta tag would still be the same.

The ideal solution is to hide all those extra variables from search engines, the only variable that should be available is the page number (when pagination is present).

This will make sure there are no multiple URLs for the same page with different variables.

This modification also includes the Multilingual URL FIX from in http://www.prestasho...ewthread/55895/

This fix will prevent Search Engines from seeing these new pages, however, if they already crawled your site and have those links, they will continue to crawl them unless told otherwise.

I have added to my Duplicate URL Redirect module 301 redirects from each page with variables (11-category-name?orderby=name&orderway=asc) to the page without any variables (11-category-name).

The only variable that will stay is p= which is used for pagination.

I have been closely monitoring the HTML suggestions on the Google webmaster tools since I made these changes and upgraded the Duplicate URL Redirect module (over 3 weeks ago), and the number of duplicate meta tags listed there went down from around 800-900 to about 100.

Attached is a zip with the modified files (4 in total) for PS 1.2, 1.3 and 1.3.2, make sure you keep a backup of your files before copying the modified ones.

HOW TO TEST?
This change does not affect regular users, so to test it, you need to trick the server to think you're a search engine.

1) Open Firefox, in the address bar type "about:config" and press enter.
2) At the top there's a filter, type "agent"
3) Right click on "general.useragent.extra.firefox" and choose modify.
4) Replace the value inside with "bot" and click OK

Now when you access your site, the module will you are a search engine, and apply the changes.
You will see that choose a filter will not do anything, the Results per page dropdown is gone, and if you enter one of those URLs with extra variables, they will all be stripped out (except p= if it was present).

To return the browser to normal, right click on "general.useragent.extra.firefox" and click on "reset".

UPDATED 11/20/10 (added /modules/sendtoafriend/sendtoafriend.php)

Attached Files

•  SEO_Fix.zip   25bytes   688 downloads

[outlet.ee]

Thank you very much!

Did I get it right that there is no change for regular user? I mean for me there seems no change, I still have urls like

/lang-ru/3_diesel?isolang=ru&n=30&id_lang=4&orderby=price&orderway=asc&p=2

not redirecting anywhere.

About the Duplicate URL Redirect module: I bought the module some time ago now, shall I buy it once more in order to get the update? Does this fix works without the Duplicate url module update?

[tomerg3]

You never have to buy a module again from my site (unless it's a new copy for a new website), when there's an upgrade and the price is the same, it's free, and if the price goes up, you just pay the difference.
To get a new version, you should email me from the same address you used to sign up on the site.

This fix works regardless of the module, it eliminates the "bad" URLs from the site.

The Duplicate URL Redirect module redirects those URLs which were already crawled and indexed by Search Engines to the main address (without the variables).

[David @ gfe.]

Tomer,

Do I need to apply this fix to the update you sent a day ago for this module? Also, do I need to edit the htaccess file any more? (I have just upgraded to 1.3.2 and so my htaccess has not been edited to accommodate this module).

Also, do i still need to edit the tools.php file as suggested here: http://www.prestasho...tag_duplication

[tomerg3]

See the previous reply for the first part of your question.

There are no changes to the htaccess file.

You should still make those changes, although the only one that's really needed is the p one (for pagination).

I will update that post in the next few days.

[refexpert]

Gud Job Mister Tomer. Your modifications are for sure usefull for the community. But my question is : Do your module hide those URLS to serch engines ?

What i am talking about is to find a solution, by the file "robots.txt" to prohibit Crawling of those pages by search engines. There are many pages concerned by the duplicate. I give you the list :

- prices-drop.php
- new-products.php
- best-sales.php
- category.php
- search.php

And maybe many other pages...

I try do it on one of my website by this. But it must be improved. So in my "robots.txt" file I add this :

# Categorie Nouveaux produits
Disallow: /new-products.php?orderby=quantity&orderway=asc
Disallow: /new-products.php?orderby=name&orderway=asc
Disallow: /new-products.php?orderby=position&orderway=asc
Disallow: /new-products.php?orderby=price&orderway=asc

# Categorie Meilleures Ventes
Disallow: /best-sales.php?orderby=quantity&orderway=asc
Disallow: /best-sales.php?orderby=name&orderway=asc
Disallow: /best-sales.php?orderby=position&orderway=asc
Disallow: /best-sales.php?orderby=price&orderway=asc

# Categorie Price Drop
Disallow: /prices-drop.php?orderby=quantity&orderway=asc
Disallow: /prices-drop.php?orderby=name&orderway=asc
Disallow: /prices-drop.php?orderby=position&orderway=asc
Disallow: /prices-drop.php?orderby=price&orderway=asc

Have to do the same for "category.php"

Tell me what you think about it. And when the job is done, i will translate your contribution.

Take care!

[refexpert]

Also, for :

Those get these urls in there sitemaps and in the INDEX of search angines, the solution must be (and your are rigtht) to do a 301 redirects. And after (months after) to delete them from Google webmasters Tools, Yahoo site explarer ........

But for new users of Prestashop, that will be Good to get sitemap.xml files whitout thousands of duplicate content. Only using the robot.txt file and the attribute "DISALLOW".

Think about it. Thank you.

Rodger

[tomerg3]

The problem with robots.txt is that search engines do not always respect them, I see them crawling pages that they are not supposed to (/cart.php is a good example).

This modification completly hides all the links you are talking about from search engines, and if they don't see them, they won't crawl them.

The redirect module from my shop redirects those URLs to the main page, which is useful for existing stores (since search engines already know a lot of those URLs)

[refexpert]

You are right for the robots.txt. I dont think about it before.

Another solution is to do the redirects 301 from the .htaccess file. But that is not the question. I will try it.

But I think your module is very very helpfull ! Really. Many people are waiting for that. So can I test it for free. And after i will give you my opinion about improvements.... and help ya for translations ti french ?

I really wanna see it working on a website.

[tomerg3]

From 1288401734:

You are right for the robots.txt. I dont think about it before.

Another solution is to do the redirects 301 from the .htaccess file. But that is not the question. I will try it.

But I think your module is very very helpfull ! Really. Many people are waiting for that. So can I test it for free. And after i will give you my opinion about improvements.... and help ya for translations ti french ?

I really wanna see it working on a website.

Many people are using it (over 200), it's installed on my site, so you can try it out, there's also a video I made on how to install and test it (it's found on the product page in my site).

To test the ?orderby=xxx redirects, you have to follow the "How to test" instructions from the first post here, and trick Firefox to identify you as a search engine (since the redirect module only does those redirects for search engines).

[noesac]

Hi Tomer, I'm also wondering...in Prestashop 1.3.2 there are some SEO fixes.

Is this part of that fix or is this actually what it is referring to?

[tomerg3]

This has not been fixed as of yet, I came up with it after 1.3.2 came out :)

[noesac]

ok thanks Tomer

[crisb]

Hi Tomer,

I did the changes you suggested on the this post: http://www.prestasho...ewthread/52665/

My doubt is should I use this fix together with the changes from that post or do i need to change that file back to original and than use the fix files?

Thanks,

[tomerg3]

It can be used together, the part that is needed is the ?p=1 page (page number), but it won't hurt keeping it there.

I need to modify it to also account for the send to a friend page, which I will in the next few days.

[uddhava]

I also noticed this "problem" on google webmaster tools. Now your suggestions are wonderful and i am indebted. But i am also a bit lost on which fix superseded other fixes you have suggested in the past and on your site.

On your site you have
1 the url redirect (nonwww to www)
2. duplicatorurl
3. meta tug duplication
4. meta tag dupl. for multi language

And now you have your latest excellent seo fix.
I understand that the duplicator module stand seperate of the seo fixes.

So which should be implemented?

[uddhava]

If you search a lot on the forum you can find others who have provided some sort of solution
that does what duplicator does. But i would trust my code only to Tomerg concerning these 'hacks'.
I like the module, and it works. And 25 dollars is peanuts.

[uddhava]

It is an easy installable module.
You only need to edit a core file and thats it.

[tomerg3]

From 1289420335:

I also noticed this "problem" on google webmaster tools. Now your suggestions are wonderful and i am indebted. But i am also a bit lost on which fix superseded other fixes you have suggested in the past and on your site.

On your site you have
1 the url redirect (nonwww to www)
2. duplicatorurl
3. meta tug duplication
4. meta tag dupl. for multi language

And now you have your latest excellent seo fix.
I understand that the duplicator module stand seperate of the seo fixes.

So which should be implemented?

I have been so busy lately that I did not get a chance to update all the pages with tips that I made.

While having them all will not hurt, some are a bit redundant.

I hope to have more time this weekend to sort those out.

[BGBK]

Just what to give my 2 cents ;-)
After installing to mods, duplicate URL has gone from 779 to 545 to 221... in about 15-20 days. Hope the down trend stays!

It really works!