Jump to content

[SOLVED] Prestashop 1.5.6.1 GoogleBot Crawl - Slow BlockLayered


Recommended Posts

some one is creating 40-50 even more session on my website with above url, which does not exist on my website therefore my sql server hang up.. 

 

they are using following range of ips for attack 66.249.74.*

 

we have installed firewall on server and when try to block this range of ips, it shows ip dones not exists..

Link to comment
Share on other sites

here are sql processes/quries stock on my sql server and creating load.

 

| 4 | scottiv1_user | localhost | scottish_shop | Query | 291 | Sending data | SELECT p.`id_product`, product_shop.`condition`, p.`id_manufacturer`, sa.`quantity`, p.`weight` , m. | | 28 | root | localhost | NULL | Query | 0 | NULL | show processlist | | 51 | scottiv1_user | localhost | scottish_shop | Query | 108 | Sending data | SELECT p.`id_product`, product_shop.`condition`, p.`id_manufacturer`, sa.`quantity`, p.`weight` , m. |

Link to comment
Share on other sites

there is some security loop hole in prestashop, a fake user is accessing a page that does not exist, which is causing my sql to overload.

here is link, what is 

http://scottishkiltshop.com/en/8-jackets?selected_filters=availability-not_available%2Fchest_size-47%2Fsleeve_length-255%2Fwaist_size-50

 

Can you explain this a little more? We take security extremely seriously, how is this a loophole?

Link to comment
Share on other sites

this is not google bot, after reading your recommendation we blocked aceess of all bots but we are still under attack..

I am using comm100 chat support software, which show certain incoming connections from same range of ips, generating such bad queries which start creating load on our server.

Link to comment
Share on other sites

I had hours of conversation with bluehost vps tier 3 level support, they asked me to do following things.. 

 

Install firewall on server

disable bad bots access

tight firewall security

 

after applying all above, they are unable to solve issue, they said, its my software issue..  

so what should i do..

Edited by Chughtai (see edit history)
Link to comment
Share on other sites

Deactivate your blocklayered, it should helps the SQL

 

blocklayered can have a pretty massive impact when the set of choice increase.

 

after disabling Layered navigation block, there is tremendous difference of load, but still hell of traffic from same range of ips

Link to comment
Share on other sites

Now your blocklayered is disabled, hence your SQL goes better

 

The IP's are not badbot, I doubt you've been talking to any tech at Lvele3 about that since:

32.74.249.66.in-addr.arpa. 86400 IN    PTR    crawl-66-249-74-32.googlebot.com.
51.74.249.66.in-addr.arpa. 86400 IN    PTR    crawl-66-249-74-51.googlebot.com.

Now you just need to edit your robots.txt, add:

*

Disallow /

 

Actually your module have filled googlebot with zillions of urls reason they can't stop trying

Configure your robots.txt to ignore properly some hash/url

 

Case closed

 

PS: Return proper http code should you be facing too high rate such as 469. Googlebot obey those btw

  • Like 2
Link to comment
Share on other sites

Now your blocklayered is disabled, hence your SQL goes better

 

The IP's are not badbot, I doubt you've been talking to any tech at Level3 about that since:

 

 

This is true. Chughtai, although you can disable it. 35 concurrent IP's on a VPS is not going to make a big difference. They are crawling your lost pages, maybe for SEO and referencing.

  • Like 1
Link to comment
Share on other sites

You welcome.

 

Regarding blocklayered it generates a lots of url for the purpose of providing some smart search mechanism.

When your search option get really wide that can fool crawlers about your content (basically they try every set of options)

You better prevent crawler from indexing them (robots.txt excluded path).

And bear in mind that layered search has also an impact on your server in regard a combinations that need to be fetched ...

 

The thumb rule is keep it simple, and have a consensus in between what you whould like to offer vs how much that impact your perf

Link to comment
Share on other sites

  • 3 months later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...