PS 1.5.4.0 - slow response - 6 cores cpu

[cogeanumarius]

Hi dear community,

I have some problems with my web-shop.

I had a really slow response from the website, so I tried to identify where the problem could be, this is what I did:

- I have updated from 1.4.9.0 to 1.5.4.0

- I have copied the website (3000 products) on a fresh install of Ubuntu 12.10 x64 with LAMP.

- I have configured Memcache: http://www.php.net/manual/ro/memcache.installation.php

- the hardware is: AMD 6 cores, 4GB RAM, HDD 2TB 5400rpm WDC green, gigabit local network

 

I have 7-10 seconds response time, from the moment I click one product, till I get the product page, any hints?

I am willing to share all my config files, just please tell me what do you need in order to suggest something

All the best, happy selling

Marius

[cogeanumarius]

Hi to all,

I have done the following tests:

1 - I have installed fresh copy of Prestashop 1.5.4.0 (default Apple products are accessible in 1 second from click on the product)

2 - When I try my website with 3000 products on this environment (products are accessible in 10 second from click on the product)

3 - I have copied from my website (3000 prod) to the fresh install just the /img directory and the mysql DB (the products are now again accessible in 10 seconds)

 

My conclusion would be that the problem resides in one of these 2 places, access of images on the HDD or accessing of the DB

Any idea of how this could be enhanced, solved?

Huge thanks in advance

PS:

I have recreated the images, and I am using the "new structure" of directories

[aduarte]

I got same problem with 24gb of ram.. going nuts to make faster.

[Tung at RockPOS.com]

Hi there,

How did you upgrade your web shop?

[aduarte]

Hi i upgraded using 1 click upgrade.

[clayton29657]

Hello CleanDev may I ask what the site address is? Maybe if you can provide this some of us here may be able to help resolve your issues

[Doulas Akula]

Am not sure of the status of your smarty templates cache but this is one thing that greatly reduces the load time for prestashop sites if it has been set to recompile templates after each invocation. The settings are found in preferences >> performance. Then the smarty section. Force compile should be set to No and cache set to Yes. Thats just one thing to ensure is right as you explore other alternatives. You may also need to contact your web host to discuss the bandwidth allowed for your site or review your hosting package.

 

Hope this helps.

[Tung at RockPOS.com]

aduarte,

Sorry to say, it believe this 1-click upgrade tool does not work for you, unless you are upgrading from a fresh installation!

 

So, in this case, there must be so many errors during executing time.

Let's turn on DEV mode to show up a list of error. Or better if you have access to your server log.

[cogeanumarius]

Hi to all of yo

Hello CleanDev may I ask what the site address is? Maybe if you can provide this some of us here may be able to help resolve your issues

The website is this one:

http://www.anticariatalbert.com/

I have disabled all the modules that I could think of, this is the reason why it looks so simple.

Thank you for any advice that I may get

[dmsims]

It is always the first Get that is slow - rest loads quickly after

 

It suggests something internal to the server

[the.rampage.rado]

It's not about the server it's about the system.

 

My installation is the same - crappy first page then everything is fine.

[cogeanumarius]

It is always the first Get that is slow - rest loads quickly after

 

It suggests something internal to the server

What I have seen it is actually that each product display is slow, more than 5 seconds, do you have the same experience?

Thank you

[mattak]

Hallo

 

I had same problem, and get it solved when my webhost disable cache on the webserver (litespeed).

 

When I upload image number 2, my screen get a grey rectangel, and have to use ctrl+f5. By next image no problem.

 

Mattak

[Kamland]

Hi

I have got the same problem using Version 1.5.3.1 - i used GTMETRIX.COM to check what happens and it's always the same:

the waiting time of getting the first byte is the problem. After receiving the first byte everything is working well.

"CleanDev" i also checked your shop with GTMETRIX - it's 100 percent the same problem like mine. It seems to be a problem all request are handled by the index.php file.

Btw. that my Shop url: www.e-zigarette-kamland.de

 

Markus

[cogeanumarius]

Hi

I have got the same problem using Version 1.5.3.1 - i used GTMETRIX.COM to check what happens and it's always the same:

the waiting time of getting the first byte is the problem. After receiving the first byte everything is working well.

"CleanDev" i also checked your shop with GTMETRIX - it's 100 percent the same problem like mine. It seems to be a problem all request are handled by the index.php file.

Btw. that my Shop url: www.e-zigarette-kamland.de

 

Markus

So, do you think there is any fix for the problem?

if you find something am also really interested.

I have tried all the possible things I could think of.

Thank you in advance

[dmsims]

You have to determine why the first get is so slow

 

It's not DNS or net traffic

 

What is the server doing during that time?

 

Have you tried a clean install of Prestashop ?

 

Have installed any mods/themes and have you tried disabling them and switching to the default theme ?

[sadlyblue]

Hi

 

As i posted on another topic:

We have a multishop, with 5 different domains, sharing about 2500 products.

Our load time is around 9-10s.

Its a dedicated server, maybe not the best one. A Intel Core i5-2400, quad core, 16 GB DDR3 RAM,...

I'm trying to get it lower, but still couldn't make significant changes.

 

On this level (many products and multishops) I think prestashop takes a little too long to get the pages ready to display. Or maybe any other software would take as long, and we need a better server

Haven't quite ficured it out...

 

Thanks

Pedro

[benjamin utterback]

Hi Pedro,

 

PrestaShop has a good loading time, that still seems like a longer than usual time. I don't think any other software would perform better. Do you have a URL?

[sadlyblue]

Hi Pedro,

 

PrestaShop has a good loading time, that still seems like a longer than usual time. I don't think any other software would perform better. Do you have a URL?

I have the same opinion. Unless we design a software for that. Maybe not so easy to integrate with modules and all, but with what we need. Not easy.

 

In the meantime, one of the domains is www.sex4funwholesale.com

I've reconfigured my.cnf for sql. And hopping it will solve the load time before first byte...

If you have any ideas, i appreciate any help..

Thanks

[cogeanumarius]

I have the same opinion. Unless we design a software for that. Maybe not so easy to integrate with modules and all, but with what we need. Not easy.

 

In the meantime, one of the domains is www.sex4funwholesale.com

I've reconfigured my.cnf for sql. And hopping it will solve the load time before first byte...

If you have any ideas, i appreciate any help..

Thanks

Salve sadlyblue,

 

may we have some details about the adjustments that you have made to "my.cnf" and "sql" so that we could try the same on our systems.

 

Huge thanks in advance

[sadlyblue]

I have this in my.cnf

 

[mysqld]

key_buffer = 512M

max_allowed_packet = 64M

table_cache = 4192

table_definition_cache = 1024

open_files_limit = 3K

sort_buffer_size = 8M

read_buffer_size = 8M

read_rnd_buffer_size = 4M

myisam_sort_buffer_size = 64M

tmp_table_size = 128M

query_cache_size = 128M

query_cache_type = 1

query_cache_limit = 4M

thread_cache_size = 16K

max_connections = 400

wait_timeout = 300

low_priority_updates = 1

concurrent_insert = ALWAYS

long_query_time = 7

log-slow-queries

 

But still get high loag times. And cpu usage for sql if of the chart.

Gonna search some more, and wait 24-48h as recommended to get to more conclusions...

[cogeanumarius]

This is something I found as really interesting, please take your time and read it all:

 

http://www.envision-systems.com.au/blog/2010/08/17/performance-tuning-a-centos-lamp-web-server-for-high-traffic-volumes/

[sadlyblue]

This is something I found as really interesting, please take your time and read it all:

 

http://www.envision-...raffic-volumes/

 

Thanks for the link.

Anyway, some of the config there only slowed the server.

I've made a few changes since i last posted my.cnf

Will try to tweak a little more and then i'll post it again.

 

One major setting missing was: innodb_buffer_pool_size = 12G

Since we have 16Gb ram, and mostly use for PS, it make sense to let most of it be used by Innodb, as i found on a search...

 

Hope to get the first received byte faster...

[benjamin utterback]

Any idea of how this could be enhanced, solved?

Huge thanks in advance

 

Well, it is strange because I ran the tool YSlow, for website speed. It gave you a really high score. It is based on number of http requests, javascripts etc..

 

It seems like it could be more server related than you previously thought.

 

[cogeanumarius]

I have this in my.cnf

...

But still get high loag times. And cpu usage for sql if of the chart.

Gonna search some more, and wait 24-48h as recommended to get to more conclusions...

salve sadlyblue,

what I have done to lower the CPU usage was to setup firewall and to block all the asian crawlers, here is the list of the IP's that I have blocked

rules.zip

[cogeanumarius]

Well, it is strange because I ran the tool YSlow, for website speed. It gave you a really high score. It is based on number of http requests, javascripts etc..

 

It seems like it could be more server related than you previously thought.

 

Salve Benjamin,

could be that on Yslow I may get good mark, but the user experience is not what I would expect, please try it yourself and browse several products, read the description and add them to your cart, you'll feel the latency each time when you press on a product

[benjamin utterback]

I did see that, but it's strange to me that it would give you a high mark (higher than most websites) but it still loads slow. That is why I'm thinking if it would be the same on another server.

[cogeanumarius]

I did see that, but it's strange to me that it would give you a high mark (higher than most websites) but it still loads slow. That is why I'm thinking if it would be the same on another server.

I have moved the website several times, on different servers on different data centers, it was hosted on

- shared environment: http://www.ixwebhosting.com/

- VPS Germany Strasbourg: http://www.server4you.com/technik/high-speed-datacenter.php

- and now on VPS USA ST. Louis also on server4you.

from any shared environment that I have tried (more than 5 till now) I got kicked out in less than a week with ToS (Terms of Service) Violation Fair Use - Web Server Overload

[benjamin utterback]

Hi, the way I'm looking it is that I've seen multiple PrestaShop stores with 4000+ products on VPS servers with as little as 4GB of ram, they load much faster than that.

 

PrestaShop only pulls a handful of products at a time, so the number of products is not going to be the sole determining factor.

[cogeanumarius]

Hi,

....

PrestaShop only pulls a handful of products at a time, so the number of products is not going to be the sole determining factor.

I perfectly understand that, please give us some useful hints that we could check, and I would gladly share with the community all the settings that I am using, and maybe we could identify where the problem resides.

Huge thanks in advance

[benjamin utterback]

Hi, This is the same website that you've used on all hosts that you get kicked out of?

[cogeanumarius]

Hi, This is the same website that you've used on all hosts that you get kicked out of?

I confirm, this is the website: http://www.anticariatalbert.com/

[Kamland]

Hi,

the problem are not the http request (69 by my webshop) these request are only take 2-3 seconds.

The problem is the waiting time for the first byte. Here you can take a look : http://gtmetrix.com/...and.de/hyY77mH3

 

Here you can check the time line of the same domain only a html file: http://gtmetrix.com/reports/www.e-zigarette-kamland.de/7gT0zodR

Waiting time is really OK of these page.

 

My Prestashop is configured "never recompile templates" but today i did some changes on a template and to my surprise i saw the changes in the FO just in time. Does that mean prestashop will always check for updated templates and scripts?

 

Also i checked the MySQL reports. There are a lot of querries which can not use an index. I got reported more than 1300 of them within 1 hour, but i don't know how to figure out which are having these index problems.

 

Markus

Edited April 25, 2013 by Kamland (see edit history)

[sadlyblue]

As i read, while searching solutions, the time untill first byt is a bit elusive or deceiving.

Since you use gzip compressions (as i do) the server as to take time to compresse it, not responding right away with data, as opposed with not using gzip compression.

But still, it's a lot of time (same problem as me).

What's the server specs? Number of stores and products you have?

[Kamland]

Only one store with 400 products.

AMD Quad-Core with 2.5 Ghz 16 GB Ram - the Server is not the problem.

Without gzip no changes - also the first byte takes a long time

Edited April 25, 2013 by Kamland (see edit history)

[sadlyblue]

That should be 2-3 seconds at most.

Try disabling some modules and see if it gets better.

Sometimes there's a module slowing things...

If not that, you may need the same as I, sql optimization. That's my guess.

[cogeanumarius]

That should be 2-3 seconds at most.

Try disabling some modules and see if it gets better.

Sometimes there's a module slowing things...

If not that, you may need the same as I, sql optimization. That's my guess.

I have disabled all the modules I could think of, the only installed and activated are: 1. Google site map, 2. reduced prices block, 3. Payment on delivery, 4. fast search, 5. e-mail alert, 6. watermark, 7. vistors online, 8. stock stats, 9. visitors and visits, 10. cart block, 11. category block, 13. CMS block, 14. Link bloc, 15. manufacturer block, 16. my account, 17. new products, 18. Permanent links, 19. Client info, 20. Promo products, 21. Products in the same category

 

I see no module that would be so demanding for my server to handle

[Trip]

Interesting thread. Unfortunatly optimising the webserver is pretty much trial and error. I am using more or less the standard apache config and recommend using mysqltuner http://www.howtoforge.com/tuning-mysql-performance-with-mysqltuner if you have root access. Using to much ram for innodb makes no sence. If your innodb is e.g. 200mb you only need to reserve 200mb. I have about 20 websites, 2 shops etc on my box and can't complain that much. I have my doubts with memcached because in my case it always slowing down the website so I do not use it and can not recommend it. The only thing which bothers me is the category view in ps. It makes a lot of queries. I think that could be optimised as in my case I only need product title and price but ps produces a lot of db queries for a relative simple task.

[dmsims]

That is pure supposition on your part

 

Disable them all and renable them one by one - testing each time

 

Maybe none of the modules is the problem - but you don't know that

 

You need to be methodical and not just guess - it's the first rule of troubleshooting

 

I have disabled all the modules I could think of, the only installed and activated are: 1. Google site map, 2. reduced prices block, 3. Payment on delivery, 4. fast search, 5. e-mail alert, 6. watermark, 7. vistors online, 8. stock stats, 9. visitors and visits, 10. cart block, 11. category block, 13. CMS block, 14. Link bloc, 15. manufacturer block, 16. my account, 17. new products, 18. Permanent links, 19. Client info, 20. Promo products, 21. Products in the same category

 

I see no module that would be so demanding for my server to handle

[sadlyblue]

About the modules, i'll do some more testing tomorrow.

In a test install i have, i'll disable everything, and delete all but one store.

Lets see how it goes...

 

Sql optimizing, i'm trying, a lot of trial and error indeed. And searching internet for some configs...

[Trip]

At cleandev. I would check the category block. In your case you could use a static html block which would save some time.

At sadlyblue. There are no configs on the internet. Every hosting is different especially concerning the database. Check out the tuning script I posted. That will tell you the parameters you should adjust. Using all the memory for innodb is not the way. Linux is designed to use as much ram as possible for caching so wasting it for the db if you don't need it is far from optimal. When you have 10 or twenty concurrent users on your server and php has a memory limit for example of 128 mb than the server is going to swap pretty fast. I recommend to use "top" to see if swap memory is used. Also munin is a nice tool to get server stats and if there are problems.

[sadlyblue]

Thanks for the advise.

What i meant for search for configs is to see what each option does, and what similar servers have set. Then i try to tweak it (up/down) each time checking the result...

[Trip]

Just found another one which seems pretty usefull for apache tuning

https://github.com/gusmaskowitz/apachebuddy.pl

[cogeanumarius]

At cleandev. I would check the category block. In your case you could use a static html block which would save some time.

...

I am not sure this is the problem, even if you highlighted that the categories are creating a lot of sql queries, my issue is with each product, not only with categories, each product takes 10 seconds to load.

[Trip]

Yepp, but on every product page the categories are rendered too. I would just compare the load time with and without cats so maybe it makes a difference and after that you can consider optimisation. Btw. did you enable debug profiling to see what hook or whatever takes so long?

[cogeanumarius]

Yepp, but on every product page the categories are rendered too. I would just compare the load time with and without cats so maybe it makes a difference and after that you can consider optimisation. Btw. did you enable debug profiling to see what hook or whatever takes so long?

Wow, I have just disable the manufacturers block and the site is 3 times faster, each product loads in 2-3 seconds.

I have 1500 manufacturers for those 3000 products

Edited April 25, 2013 by CleanDev.eu (see edit history)

[sadlyblue]

Wow, I have just disable the manufacturers block and the site is 3 times faster, each product loads in 2-3 seconds.

I have 1500 manufacturers for those 3000 products

 

Nice to know. I'll try it myself and let you know.

I've also heard that the categories block is slows the site down. But i've disabled it once, and it changed nothing (same load time).

[cogeanumarius]

Nice to know. I'll try it myself and let you know.

I've also heard that the categories block is slows the site down. But i've disabled it once, and it changed nothing (same load time).

same here,

disabling the categories block improved nothing, but disabling the manufacturers did a noticeable faster loading for each product.

[Kamland]

SQL optimizing will not work if there are full table scans working because of missing indexes.

Now I will check the hole database if i can find tables with not existing indexes.

Btw. someone knows why are no RI used inside the prestashop database?

[Kamland]

As I wrote i checked the database if any index is missing. Just found some table primary used in BO queries.

What I also did I disabeled the memcache function and time getting the first byte back was reduced of 1 second. And the hole start of the shop is about 1.5 seconds faster.

Disabling the manufacturer improved nothing for me.

Btw. "CleanDev" your shops are really fast now.

[cogeanumarius]

...

Btw. "CleanDev" your shops are really fast now.

I have 3000 products and 1500 manufacturers, maybe this is the reason why disabling the manufactures had some noticeable effect on my shop.

[sadlyblue]

I've tried memcahed, xcache, filesystem. All worked almost the same...

 

Strangest thing, i disabled the cache, forced compilation,... guess what, i got almost the same results.

So it's not the cpu/sql/ram problem anymore. Something is slowing down prestashop. If i open an html file, it's fast.

And if it can get the full page at the speed with cache on or off it can't be the server specs problem, right?

[Kamland]

Yes you are right something is slowing down prestashop. That must be a routine, which will not only run on the entry site of the shop because also the product pages and category pages are slow. All the CMS pages seem to be 30 percent faster.

Maybe a member of the prestashop developing team is able to tell me which modules or procedures will always run again with loading a new page.

[sadlyblue]

A few days have passed but no improvement...

And i haven't even narrowed down the problem. I think prestashop isn't fast enought with the kind of usage we need. Although we don't have that many users online (we sell to stores only) our database has been growing and maybe as passed the limit that prestashop can handle fast enough.

Some errors in error_log don't help in performance. Mainly unlink errors.

And some features, hooks, and others, that we don't need also help slow things down.

 

I'm seriously thinking in redisigning the front office for our usage... I know it would take a lot of work. But at least it would clear things. If its a Prestashop problem or not...

I believe it is. With a brand new install it worked quite fast, untill i started importing so many products...

Our server, not being the fastest, i think is capable of handling what we need.

 

Another thing we are going to try before that is switching servers. Not to a different one, to one with the same specs, but brand new install... I know its a lot of work, but far less that the other thing, and i get the partitioning we should have (more root size), and other software cleaned...

 

If someone has any better ideas, let me know...

[benjamin utterback]

Hi, just wondering if you have seen how fast it is locally. It's not a "real, live" environment but it may give you a better idea.

[Trip]

Without having access to your site, ssh access etc.pp. we can only guess the bottleneck. I mean CleanDev.eu is the best example. How can we possibly know that he has 1500 manufacturers. That possibly means minumum 1500 (extra)-database queries, maybe with joins but than it makes sense that with a bad optimized database server the performance is horrible. Usually the debug profiling in defines.inc.php can show you if there is heavy use off the db or any hook and which is blocking execution time. So far I only can tell you that my international shop in the footer is running 1.5.4.1 and the german 1.4.10 and the performance it nearly the same.

Edited April 30, 2013 by Trip (see edit history)