Super High Server Load | Pollin

[Gregg007]

Hi Everyone, 

 

I'm not new to Prestashop, but new around here, 

I find myself in a situation where I cannot figure out as to why the server of which my website is sitting, continues to sky rocket the servers load and when checking the pID's the following is being shown, 

brk(0)                                  = 0x7f059f34e000
brk(0)                                  = 0x7f059f34e000
brk(0x7f059f18e000)                     = 0x7f059f18e000
brk(0)                                  = 0x7f059f18e000
sendto(3, "\262\0\0\0\3\n\t\t\t\tSELECT `priority`, `id"..., 182, MSG_DONTWAIT, NULL, 0) = 182
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\1\0\0\1\2g\0\0\2\3def\rsimplici_live\32ps_s"..., 9874, MSG_DONTWAIT, NULL, NULL) = 273
sendto(3, " \3\0\0\3\n\t\t\tSELECT *, ( IF (`id_gro"..., 804, MSG_DONTWAIT, NULL, 0) = 804
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\1\0\0\1\23g\0\0\2\3def\rsimplici_live\21ps_s"..., 9601, MSG_DONTWAIT, NULL, NULL) = 1875
sendto(3, "w\2\0\0\3SELECT product_shop.`price`"..., 635, MSG_DONTWAIT, NULL, 0) = 635
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\1\0\0\1\6H\0\0\2\3def\rsimplici_live\fprod"..., 7726, MSG_DONTWAIT, NULL, NULL) = 649
sendto(3, "{\0\0\0\3\n\t\t\t\t\t\t\tSELECT `id_tax_rule"..., 127, MSG_DONTWAIT, NULL, 0) = 127
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\1\0\0\1\1e\0\0\2\3def\rsimplici_live\17ps_p"..., 7077, MSG_DONTWAIT, NULL, NULL) = 134
sendto(3, "|\0\0\0\3\n\t\t\tSELECT `reduction`\n\t\t\tF"..., 128, MSG_DONTWAIT, NULL, 0) = 128
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\1\0\0\1\1u\0\0\2\3def\rsimplici_live ps_p"..., 6943, MSG_DONTWAIT, NULL, NULL) = 144
sendto(3, "\207\0\0\0\3\n\t\tSELECT t.`id_lang`, t.`n"..., 139, MSG_DONTWAIT, NULL, 0) = 139
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\1\0\0\1\0028\0\0\2\3def\rsimplici_live\1t\6ps"..., 6799, MSG_DONTWAIT, NULL, NULL) = 137
sendto(3, "\234\0\0\0\3SELECT SUM(quantity)\nFROM `"..., 160, MSG_DONTWAIT, NULL, 0) = 160
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\1\0\0\1\1#\0\0\2\3def\0\0\0\rSUM(quantity)\0\f"..., 6662, MSG_DONTWAIT, NULL, NULL) = 68
sendto(3, "\233\0\0\0\3SELECT out_of_stock\nFROM `p"..., 159, MSG_DONTWAIT, NULL, 0) = 159
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\1\0\0\1\1_\0\0\2\3def\rsimplici_live\22ps_s"..., 6594, MSG_DONTWAIT, NULL, NULL) = 128
sendto(3, "\237\0\0\0\3SELECT depends_on_stock\nFRO"..., 163, MSG_DONTWAIT, NULL, 0) = 163
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\1\0\0\1\1g\0\0\2\3def\rsimplici_live\22ps_s"..., 6466, MSG_DONTWAIT, NULL, NULL) = 136
sendto(3, "\354\0\0\0\3SELECT image_shop.`id_image"..., 240, MSG_DONTWAIT, NULL, 0) = 240
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\1\0\0\1\1J\0\0\2\3def\rsimplici_live\nimag"..., 6330, MSG_DONTWAIT, NULL, NULL) = 112
access("/home/idshosting/public_html/img/p/254739-128420-5.jpg", F_OK) = -1 ENOENT (No such file or directory)
access("/home/idshosting/public_html/img/p/1/2/8/4/2/0/128420-5.jpg", F_OK) = -1 ENOENT (No such file or directory)
brk(0)                                  = 0x7f059f18e000
brk(0x7f059f1ce000)                     = 0x7f059f1ce000
brk(0)                                  = 0x7f059f1ce000
brk(0x7f059f20e000)                     = 0x7f059f20e000
brk(0)                                  = 0x7f059f20e000
brk(0x7f059f24e000)                     = 0x7f059f24e000

I have tried changing server configurations, but nothing seems to help, 

 

Prestashop version - 1.6.1.10
No modifications to the core (That I'm aware of)

Server Details:

• 8vCPU's
• 16GB RAM

 

Website link - https://simplicity.co.za

 

If anyone can assist or at least point me in the right direction, 

 

Thanks

 

[Scully]

It's a little vague information in order to help. What poll concerns:

 

https://stackoverflow.com/questions/13018981/poll-system-call-timeout

 

https://stackoverflow.com/questions/24791625/how-to-handle-the-linux-socket-revents-pollerr-pollhup-and-pollnval

 

Whant general activity concerns:

• Have you already restarted the entire server?
• How is load developing after restart?
• Did you check how many visitors you have in your statistics?
• Did these figures increase recently?
• Did you check the top 5 IP addresses which access your server?
• Are they known to you or at least explainable?
• Did you check unknown IP's accessing your backoffice URL?
• Did you check recentrly changed files within your store? (do not count for cache files)
  find with -mtime would be your friend to do so.

Edited August 28, 2017 by Scully (see edit history)

[selectshop.at]

How many websites are you hosting on the server ? Piped Logs activated ? First of all you should discover which file is causing the error.

[Gregg007]

This particular server is only hosting this site, 

From the Process manager within WHM, its always this particular accounts USERNAME which is running, 

 

I've been able to confirm that the increased traffic, increases the number of these processors running BUT, on another server we have a Prestashop installed, Same version and it gets much higher traffic without using as much server resources, 

 

 

idshosting in this case is the simplicity account running, 

 

As you can see the first 3 lines are the same script running and this can continue to over 30 times - Something isn't finishing and its starting again, 

 

@selectshop - Piped logs? Please elaborate, 

 

Thanks

[selectshop.at]

piped logs do not take effect, if your site is alone on the server. Did you already checked broken tables on your database ? If you trace pid there is no hint or viable information about which process/file is creating the overload ?

[selectshop.at]

1) You can test Prestashop database loads by activating debug mode on /config/defines.inc.php file (activate SQL debug as well). Turn shop to under maintenance. Test site by yourself and see where you are having bottle necks on your theme in use. Test also back-office with debug mode activated. Perhaps you have some extra modules opening processes.

 

2) Check database for broken tables.

[Gregg007]

@selectshop - How would I check for broken tables? 

 

Thanks

[Gregg007]

Current situation - I've started killing off PID's, 

 

[selectshop.at]

1) Killing processes aren't the solution. With phpMyAdmin you can check for broken tables and repair them, see tutorial here: http://www.inmotionhosting.com/support/website/databases/check-repair-database

 

2) For to see which script is causing the problem you should use Prestashop debug mode (all options enabled)

/* Debug only */
if (!defined('_PS_MODE_DEV_')) {
define('_PS_MODE_DEV_', true);
}
/* Compatibility warning */
define('_PS_DISPLAY_COMPATIBILITY_WARNING_', true);
if (_PS_MODE_DEV_ === true) {
    @ini_set('display_errors', 'on');
    @error_reporting(E_ALL | E_STRICT);
    define('_PS_DEBUG_SQL_', true);
} else {
    @ini_set('display_errors', 'on');
    define('_PS_DEBUG_SQL_', false);
}

Than visit your page on front and back and make some changes, open sites, etc. Mainly I think that problem is on front, so see if there are errors or bottle necks on front-office. The more options you try the more you can see performances problems on your page.

 

3) Please check also server error logs if not yet done.

[Gregg007]

Great - Thanks - Database has been checked and reported nothing out of the ordinary, 

 

I'll enable debug and see what comes up, 

 

Thanks

[selectshop.at]

One important thing is also to check how many users are visiting your page as Scully told you.

Than you should check error logs of server and server settings for time restrictions. time-outs could as well hang processes.

If you don't find nothing abnormal with Prestashop debug mode (bottle-necks on scripting), than you should revise server settings for restrictions and wrong rules set in this case.

[Scully]

Yessss - i'd still check the IP addresses leading to most traffic on my server and figure out wether they are visitors, customers, search engines or unwanted bots. If you need an SQL, just tell me. Otherwise you could also look trough your access and / or error logfiles and check the ip addresses.

[Gregg007]

Just a quick update - Seems we had an IP banging away at the server - Blocked the IP and server resources today have been quite stable, 

[Mon Aug 28 22:22:31.018841 2017] [cgi:warn] [pid 36477] [client 41.149.121.131:10186] AH01220: Timeout waiting for output from CGI script /usr/local/cpanel/cgi-sys/ea-php56
[Mon Aug 28 22:22:52.437248 2017] [cgi:warn] [pid 36461] [client 41.149.121.131:5607] AH01220: Timeout waiting for output from CGI script /usr/local/cpanel/cgi-sys/ea-php56
[Mon Aug 28 22:24:11.036840 2017] [cgi:warn] [pid 37165] [client 41.149.121.131:16265] AH01220: Timeout waiting for output from CGI script /usr/local/cpanel/cgi-sys/ea-php56

Everything is still as it should be, Will keep monitoring it, 

 

Thanks for all of your input(s)

[Scully]

Sounds very good and this was what I exactely meant with checking the IPs with the recent highest frequency. We run these checks automatically every hours and if IPs pop up with very unusual high frequency, we block them automatically. Furthermore we have a lot of county top level domains blocked with .htaccess.

 

Why: Most of our customers do not make worldwide business. But dirty bots do so.

 

You might want to add 'SOLVED' to your topic title if the problem is solved for sure.

Edited August 29, 2017 by Scully (see edit history)

[Gregg007]

Thanks @scully - I'll mark is solved in the next few hours - I'd like to monitor it for a bit, 

 

Yeah, our servers get bombarded regularly - Lots of blocked notifications coming in daily.

 

Thanks again

[selectshop.at]

Be careful, the IP you are blocking is a ZA Telkom. Could be a service you are using and not an user.  I cannot see any entries for this IP in honeypot or other spam-blocking list and they are very reliable and live dated. You should clear who is using the IP : 41.149.121.131. This IP is used by a mailserver from what I can see.

 

https://www.robtex.com/ip-lookup/41.149.121.131

 

Your server is sending unsolicited mails ? Could be a hacked script...

 

The blocking of IP results in a timeout on one of your scripts on your server. So indeed a server problem, or a not correct working script.

[selectshop.at]

Perhaps you are having this PS bug, as you are using an older PS-Version. http://forge.prestashop.com/browse/PSCSX-9132. This bug was available in PS 1.5. and PS 1.6. equally.

[Gregg007]

@selectshop - We see the same on this side, 

 

We aren't able to figure out whom the IP belongs to or where and why its happening, 

We've checked all functionality at this stage including incoming and outgoing servers without failure, 

 

I guess at this point, its whom ever shouts at this stage we'll know whom it belongs too - Its really not worth the time searching high and low, 

 

Thanks for taking a further look though 

[Gregg007]

@SelectShop - That could very well be the case, 

I'll add that to the list of items to implement, 

 

Thanks

[selectshop.at]

The IP you see on the log line. It is: 41.149.121.131

 

Your problem is not solved, you have done a cosmetic trick.

 

Be careful, the IP you are blocking is a ZA Telkom. Could be a service you are using and not an user.  I cannot see any entries for this IP in honeypot or other spam-blocking list and they are very reliable and live dated. You should clear who is using the IP : 41.149.121.131. This IP is used by a mailserver from what I can see.

 

https://www.robtex.com/ip-lookup/41.149.121.131

 

Your server is sending unsolicited mails ? Could be a hacked script...

 

The blocking of IP results in a timeout on one of your scripts on your server. So indeed a server problem, or a not correct working script.

 

 

Perhaps you are having this PS bug, as you are using an older PS-Version. http://forge.prestashop.com/browse/PSCSX-9132. This bug was available in PS 1.5. and PS 1.6. equally.

 

[selectshop.at]

Well with robtex I can see that the IP is already registered on some spam-lists. Which is the DNS of your mail server ? If it isn't equal to 41.149.121.131 you can lay back. If you are using this IP, so you should take consequences. Perhaps your server is sending unsolicited spam mails to others and this is also a reason for high traffic.

[selectshop.at]

Some update: you can lay back. The IP's of your mailserver in use are others. The IP numbers are 129.232.138.235 (used by four host names) and 196.40.97.190 (used by 46 host names). The only thing you should check if your are using external services which could use the IP 41.149.121.131. If not could be some attack to your SERVER corefiles. Take a look into your server logs when the high traffic was happening. You will find there logs from what was tried to execute. If your server didn't rejected the request, it will not be registered in the error logs. In this case only the log files can give more information about it. Search for the IP in question: 41.149.121.131