The Ultimate Cache & Performance Config

[koskimaa]

Hi all,

 

While building my store, I've had a hard time finding information on getting the best performance out of Prestashop. A lot of information is outdated, and there are many conflicting opinions. Below is a config that I imagine is the best you can get but I want you to correct me so that I can improve it in order for everyone to see and learn!

 

Server config:

 

PHP Version: 7.x.x

Caching: opcache

Cache module: JPresta Page Cache

 

Prestashop Performance settings:

 

Smarty

Template compilation: Never recompile template files

Cache: Yes

Caching type: File System

Clear cache: Clear cache everytime something has been modified

 

CCC

Smart cache for CSS: YES

Smart cache for Javascript: YES
Minify HTML: NO

Compress inline Javascript in HTML: YES

Move Javascript to the end: YES

Apache optimization: YES

 

Ciphering

Algorithm: Rijndael with mcrypt.lib

 

Caching

Use cache: NO (opcache makes this alternative obsolete)

 

CDN

 

Should you use a CDN like Cloudflare? YES, if your customers visit your shop from several different continents in the world AND/OR if your shop is very image heavy. NO, if your target customers are all from the same country or perhaps continent. Use a server in that area and configure well and a CDN is unnecessary.

 

Tracking/Analytics

 

Turn OFF all Prestashop tracking, stats and analytics modules. Use external tracking like Google Analytics or Piwik.

 

Other advanced options

 

- "Warming up" the cache

- Optimizing/cleaning the database

 

 

Please post your opinions/corrections and/or other suggestions so I can update the post and we can all get a better, more up-to-date picture on how to get the ultimate Prestashop performance!

 

[koskimaa]

This is the best article I've read so far on this topic: https://canonicalized.com/prestashop-speed-optimization/ but I don't understand which "File system" cache they refer to when they tell you to turn it off. The File system cache under Smarty or under "Caching"?

 

From what I understand, using Prestashop with PHP7 and opcache gives you the fastest base performance available. Correct?

 

Recently my host (Siteground) updated from PHP 7.0.8 to 7.0.9 which finally makes my store work with PHP7. My host also has opcache activated by default if you're using PHP7.

 

Activating PHP7 gave me a significant performance boost, and using dh42's free opcache manager module I can confirm that opcache is indeed active.

 

However, I still can't hit the magic "under 200ms" server response time/TTFB. I got closer, but still not there. Should I use a caching module or is there no point with opcache?

 

Other questions where clear answers would benefit the community:

 

- Should you always have the Smarty cache enabled in a live environment?

 

- What is the differences between memcache and memcached?

[JPresta.com]

- Should you always have the Smarty cache enabled in a live environment?

Sure, why not?

 

Also why don't you turn on HTML minifier?

[koskimaa]

Sure, why not?

 

Also why don't you turn on HTML minifier?

Since it's slow and outdated according to this post.

 

Do you disagree? In that case, please explain why so we can all learn.

 

Also, since you're the creator of the JPresta Page Cache module that I mention in my first post: Will there be any significant improvement in performance if I use your module with my store, considering I already have PHP7 and opcache active?

 

Even though I wrote about it in my first post, I haven't actually bought and tried it yet. I'd like to know before I invest.

 

What is the ultimate performance setup according to you? What should a shop owner do in order to reach that golden under 200ms TTFB/server response time + under 1 sec total page load.

 

Can you have too many layers of cache? (Smarty cache + opcache + cache module + hosting provider cache + CDN cache etc. etc..)

Edited August 4, 2016 by koskimaa (see edit history)

[PrestaHeroes USA]

The key measurement is not so much entire page load, but 'above thd fold', that is the most important measurement.

 

Measuring a shop using online metrix...pingdome, gmetrix, etc. while it provides useful information does not currently measure AFT.

The AFT is so important that Google patented it.

 

Learn more here:http://www.google.co...s/US20070118640

[PrestaHeroes USA]

p.s. on my server, all my shops without touching them, report in summary of PS profiler, 'that a boy, that's what I call a server'....lol...happy day....el

[JPresta.com]

Since it's slow and outdated according to this post.

 

Do you disagree? In that case, please explain why so we can all learn.

Outdated is not good reason. I understand that minifying the HTML takes time but I guess that the browser will parse it faster. When using an HTLM cache I think you should enable it. If not it can be faster without it.

 

 

Also, since you're the creator of the JPresta Page Cache module that I mention in my first post: Will there be any significant improvement in performance if I use your module with my store, considering I already have PHP7 and opcache active?

It will always be faster because sending an already generated page is faster than creating it everytime. Also even with a very powerfull server, the browser cache will always be faster :-)

 

 

What is the ultimate performance setup according to you? What should a shop owner do in order to reach that golden under 200ms TTFB/server response time + under 1 sec total page load.

See this post

 

 

Can you have too many layers of cache? (Smarty cache + opcache + cache module + hosting provider cache + CDN cache etc. etc..)

They all provide different level of cache that can work together

[koskimaa]

The key measurement is not so much entire page load, but 'above thd fold', that is the most important measurement.

 

Measuring a shop using online metrix...pingdome, gmetrix, etc. while it provides useful information does not currently measure AFT.

The AFT is so important that Google patented it.

 

Learn more here:http://www.google.co...s/US20070118640

 

Thanks for the input. Yes, above the fold indeed seems important. I do believe server response time and page load are equally important, not just for SEO but for user experience as well. In this day and age people are so inpatient and waiting more than 3 seconds for a page to load means they close the window and look elsewhere. That's why I think hitting that under 200ms TTFB/server response time is so crucial.

[koskimaa]

Outdated is not good reason. I understand that minifying the HTML takes time but I guess that the browser will parse it faster. When using an HTLM cache I think you should enable it. If not it can be faster without it.

 

 

It will always be faster because sending an already generated page is faster than creating it everytime. Also even with a very powerfull server, the browser cache will always be faster :-)

 

 

See this post

 

 

They all provide different level of cache that can work together

 

Thanks for the response, link and your opinions! By the way, I saw you offered a 30 day trial for your module. I'll consider trying it out. I just need to figure out if I just focus on setting up a CDN before I start with a caching module, or the other way around...

 

Also, according to you, what does your module do better than other, cheaper caching modules like this one for example?

[JPresta.com]

Also, according to you, what does your module do better than other, cheaper caching modules like this one for example?

This one does not cache pages for logged in people

This one does not use browser cache

[koskimaa]

This one does not cache pages for logged in people

This one does not use browser cache

Got it. Thanks for the response!

[Scully]

Our experience with prestashop 1.5.6. running now since 2 1/2 years:
 
The biggest culprit are the database selects. In terms of SQL, they are beautiful. One larger select fetches all data required for displaying the appropriate content.
The downside is performance. In most of the issues we tackled down to the last byte we found doing two selects instead of one reduces response time drastically.
 
How to optimize?
The trick is to enable profiling via config settings and then analyse SQL statments written out at the end of the page.
Open the php files which are indicated by the first queries. Analyze the SQL statement and make two out of one select.
 
The first select only joins tables really needed fulfill criteria for the WHERE clause which is mostly half of the tables in the large select.
So you need all the tables which are references in the WHERE clauses and all the columns which are used in the ORDER BY.
 
The first select needs also the product_id column. This column is later used in the 2nd query which fetches only the products (or categories) which are really needed.

 

This is an example for the Products by Manufacturer function (public static function getProducts) in classes / manufacturer.php. Around line 328 you see the main select like this:
 
 

$sql = 'SELECT p.*, product_shop.*, stock.out_of_stock, IFNULL(stock.quantity, 0) as quantity, MAX(product_attribute_shop.`id_product_attribute`) id_product_attribute,
                    pl.`description`, pl.`description_short`, pl.`link_rewrite`, pl.`meta_description`, pl.`meta_keywords`,
                    pl.`meta_title`, pl.`name`, MAX(image_shop.`id_image`) id_image, il.`legend`, m.`name` AS manufacturer_name,
                    DATEDIFF(
                        product_shop.`date_add`,
                        DATE_SUB(
                            NOW(),
                            INTERVAL '.(Validate::isUnsignedInt(Configuration::get('PS_NB_DAYS_NEW_PRODUCT')) ? Configuration::get('PS_NB_DAYS_NEW_PRODUCT') : 20).' DAY
                        )
                    ) > 0 AS new
                FROM `'._DB_PREFIX_.'product` p
                '.Shop::addSqlAssociation('product', 'p').'
                LEFT JOIN `'._DB_PREFIX_.'product_attribute` pa
                    ON (p.`id_product` = pa.`id_product`)
                '.Shop::addSqlAssociation('product_attribute', 'pa', false, 'product_attribute_shop.`default_on` = 1').'
                LEFT JOIN `'._DB_PREFIX_.'product_lang` pl
                    ON (p.`id_product` = pl.`id_product` AND pl.`id_lang` = '.(int)$id_lang.Shop::addSqlRestrictionOnLang('pl').')
                LEFT JOIN `'._DB_PREFIX_.'image` i
                    ON (i.`id_product` = p.`id_product`)'.
                Shop::addSqlAssociation('image', 'i', false, 'image_shop.cover=1').'
                LEFT JOIN `'._DB_PREFIX_.'image_lang` il
                    ON (i.`id_image` = il.`id_image` AND il.`id_lang` = '.(int)$id_lang.')
                LEFT JOIN `'._DB_PREFIX_.'manufacturer` m
                    ON (m.`id_manufacturer` = p.`id_manufacturer`)
                '.Product::sqlStock('p', 0).'
                WHERE p.`id_manufacturer` = '.(int)$id_manufacturer.'
                '.($active ? ' AND product_shop.`active` = 1' : '').'
                '.($front ? ' AND product_shop.`visibility` IN ("both", "catalog")' : '').'
                AND p.`id_product` IN (
                    SELECT cp.`id_product`
                    FROM `'._DB_PREFIX_.'category_group` cg
                    LEFT JOIN `'._DB_PREFIX_.'category_product` cp
                        ON (cp.`id_category` = cg.`id_category`)'.
                    ($active_category ? ' INNER JOIN `'._DB_PREFIX_.'category` ca ON cp.`id_category` = ca.`id_category` AND ca.`active` = 1' : '').'
                    WHERE cg.`id_group` '.$sql_groups.'
                )
                GROUP BY product_shop.id_product
                ORDER BY '.$alias.'`'.bqSQL($order_by).'` '.pSQL($order_way).'
                LIMIT '.(((int)$p - 1) * (int)$n).','.(int)$n;

        $result = Db::getInstance(_PS_USE_SQL_SLAVE_)->executeS($sql);

 

[Scully]

This query fetches all columns for products from a certain manufacturer. On our server each execution took around 400 ms (0.5 sec). We made two queries out of one. The first part looks now like this:

 

        // 07.10.2016 new Preselect for improved Performance    
        $sql = 'SELECT p.id_product, ps.quantity
                FROM `'._DB_PREFIX_.'product` p
                LEFT JOIN `'._DB_PREFIX_.'product_sale` ps ON p.id_product = ps.id_product
                WHERE p.`id_manufacturer` = '.(int)$id_manufacturer.'
                '.($active ? ' AND p.`active` = 1' : '').'
                '.($front ? ' AND p.`visibility` IN ("both", "catalog")' : '').'
                AND p.`id_product` IN (
                    SELECT cp.`id_product`
                    FROM `'._DB_PREFIX_.'category_group` cg
                    LEFT JOIN `'._DB_PREFIX_.'category_product` cp
                        ON (cp.`id_category` = cg.`id_category`)'.
                    ($active_category ? ' INNER JOIN `'._DB_PREFIX_.'category` ca ON cp.`id_category` = ca.`id_category` AND ca.`active` = 1' : '').'
                    WHERE cg.`id_group` '.$sql_groups.'
                )
                GROUP BY p.id_product
                ORDER BY '.$alias.''.$order_by.' '.pSQL($order_way).'
                LIMIT '.(((int)$p - 1) * (int)$n).','.(int)$n;
               
                
        $products = Db::getInstance(_PS_USE_SQL_SLAVE_)->executeS($sql);

 

 

As a result we ONLY get the id_products columns and all the ones needed for sorts.

[Scully]

This product ids given back are then processed in PHP like this:

 

        $ids = array();
        foreach ($products as $product)
        if (Validate::isUnsignedId($product['id_product']))
        $ids[$product['id_product']] = 1;

        $ids = array_keys($ids);
        $ids = array_filter($ids);
        sort($ids);
        $ids = count($ids) > 0 ? implode(',', $ids) : 'NULL';

 

 

This gives us as list like (234, 244, 256, 289, 300, 303, 304)

[Scully]

And the final SQL select which fetches all data needed now looks like this:

 

        $sql = 'SELECT p.*, product_shop.*, stock.out_of_stock, ps.quantity as quantity, MAX(product_attribute_shop.`id_product_attribute`) id_product_attribute,
                    pl.`description`, pl.`description_short`, pl.`link_rewrite`, pl.`meta_description`, pl.`meta_keywords`,
                    pl.`meta_title`, pl.`name`, MAX(image_shop.`id_image`) id_image, il.`legend`, m.`name` AS manufacturer_name,
                    p.is_new AS new
                FROM `'._DB_PREFIX_.'product` p
                '.Shop::addSqlAssociation('product', 'p').'
                LEFT JOIN `'._DB_PREFIX_.'product_sale` ps ON ps.`id_product` = p.`id_product`
                LEFT JOIN `'._DB_PREFIX_.'product_attribute` pa
                    ON (p.`id_product` = pa.`id_product`)
                '.Shop::addSqlAssociation('product_attribute', 'pa', false, 'product_attribute_shop.`default_on` = 1').'
                LEFT JOIN `'._DB_PREFIX_.'product_lang` pl
                    ON (p.`id_product` = pl.`id_product` AND pl.`id_lang` = '.(int)$id_lang.Shop::addSqlRestrictionOnLang('pl').')
                LEFT JOIN `'._DB_PREFIX_.'image` i
                    ON (i.`id_product` = p.`id_product`)'.
                Shop::addSqlAssociation('image', 'i', false, 'image_shop.cover=1').'
                LEFT JOIN `'._DB_PREFIX_.'image_lang` il
                    ON (i.`id_image` = il.`id_image` AND il.`id_lang` = '.(int)$id_lang.')
                LEFT JOIN `'._DB_PREFIX_.'manufacturer` m
                    ON (m.`id_manufacturer` = p.`id_manufacturer`)
                '.Product::sqlStock('p', 0).'
                WHERE p.`id_manufacturer` = '.(int)$id_manufacturer.'
                '.($active ? ' AND product_shop.`active` = 1' : '').'
                '.($front ? ' AND product_shop.`visibility` IN ("both", "catalog")' : '').'
                AND p.`id_product` IN ('.$ids.')
                GROUP BY product_shop.id_product
                ORDER BY '.$alias.''.$order_by.' '.pSQL($order_way); //.'
                // LIMIT '.(((int)$p - 1) * (int)$n).','.(int)$n;
                
        $result = Db::getInstance(_PS_USE_SQL_SLAVE_)->executeS($sql);

 

 

Notes:

1. Instead of doing the complex subquery again, we only fetch data from the product tables with ids coming from the pre select above.

2. The limit is no longer necessary since we limit the query already in the first select. This saves much processing time.

 

Result:

Before change: appx. 400 ms

After chane: appx. 50 ms

Saved time: 80%

[Scully]

While this example shows an original query which is rather fast in terms of prestashop, most can be optimized in categories. There we had some categries with 10 sub categories and appx. 2-3 sub-sub categories. Opening this pages needed 7 to 10 seconds to response.

After optimizing the SQL statements in the same manner as described above, the same result within less than 1 second.

 

You can see it here on our productional shop:

https://mit100wir.ch/wirshop/

[Scully]

The second improvement beyond database selects was to use the express cache module. This module saves pages in static html and merges this static pages with some dynamic data which can be defined by the user. Depending on how fast or slow your server runs, this module can save up between 20% to 60% of response time.

 

Even if we found the express cache very helpful we did also some improvements there.

 

We run a script which calls the oldest pages from the cache table to avoid expiration. This script runs basically all day long. Why do that? If pages are requested frequently, the never run into expiration.

[Scully]

And the third improvement also goes into database. We found that there was to much data in connections_source and some other tables.

So we purge some data after some days and we purge all data which is traffice from abroad.

Since we only sell in one country, we are not interested in statistics for users which would never be customers.

 

So what we delete in overview:

 

statssearch 

all after 120 days or less than 4 characters

connections_source 

all after 120 days or immediate if no helpful data

guest

if no connections id found and id_customer is 0

 

Doing this deletions reduced our database size by about 20 to 30%.

[Scully]

And the last improvement was database buffers, sort buffers and similar. This can be done by either editing the my.cnf in the respective folder OR if you don't have privilege to to so by doing this on the fly. Variables we changes for a server with 3 GB of memory and 250 MB of database size:

 

            "SET GLOBAL query_cache_size = 50331648;       # 48 MB",
            "SET GLOBAL join_buffer_size = 262144;         # 256 KB",
            "SET GLOBAL key_buffer_size = 2097152;         #   2 MB",
            "SET GLOBAL query_cache_limit = 786432;        # 768 KB",
            "SET GLOBAL query_cache_min_res_unit = 2048;   # 2KB",
            "Set Global myisam_sort_buffer_size = 2097152; # 2 MB",
            "Set Global sort_buffer_size = 4194304;        # 4 MB",
      //    "Set Global innodb_sort_buffer_size = 4194304; # 4 MB",        // as per MySQL 5.6
            "Set Global key_buffer_size = 1048576;         # 1 MB",
            "SET Global table_open_cache = 1300;",
            "SET Global table_definition_cache = 1300;";

 

This setting results in:

Query cache hit rate of 85%   1)

INNODB Read buffer efficiency of 99.9%

 

1) What is the SQL Query cache

If the same statement has been queried already before, insted of running the query againgst the sql server, only the result for the earlier query is sent back. If you only have rare traffic on your shop you might have seed that opening the start page on a monday morning for the firs time takes much more time than the 2nd call for the same page. This is also caused by a cold query cache.

In our configuration we have 48 MB of query cache. And we only allow query results to put in cache if the result has a minimum length of 2 KB and a max size of 768 KB.

These values were determine in tests. The help to avoid much query cache prunes due to low memory.

 

2) What is the Read Buffer

The read buffer holds data (tables as well as indexes) in memory as much as possible. This avoids disk access if the tables used in a query are hold in buffer tables.

The configuration differs depending if you use MyISAM or INNODB engine but the mechanism is the same.

In our example we have a buffer size of 128 MB.

 

3) What is the Sort Buffer?

The sort buffer describes how much data may be hold in memory for sorting instead of writing a temporary file for doing the sort. If an SQL select produceds more data than sort buffer is available, it will create a temp file and this increases response time.

[Scully]

Conclusion:

Optimize the SQL statements on by one, best starting with the category class.

Reduced response time by: 50 to 90%

 

Installing express cache module:

Reduced response time by: 20 to 60% (in our case is more 20% because of other optimizations coming into effect)

 

Fetching cached pages frequently

Helps to avoid expiration of cached pages. AND it helps to keep the database cache warm (filled) since the expiration only takes effect for rarely used pages.

 

Reducing database size

Helps to increase cache hit rate from database

 

Optimize MySQL parameters

Helps to increase query cache hit rate, buffer hit rate and avoids unnecessary temporary file creation on disk

Edited October 8, 2016 by Scully (see edit history)

[Scully]

To anser some question from the thread starter:

 

- Yes, always use smarty in a productional environment.

- We don't have PHP 7 so far. Our PHP is 5.5.9.

- We have 2 media servers set which run on the same server with different domains.

- We don't minify the HTML. According to our tests, if does not help in terms of performance but sometimes results in (smaller) errors on some browsers.

[PrestaHeroes USA]

To anser some question from the thread starter:

 

- Yes, always use smarty in a productional environment.

- We don't have PHP 7 so far. Our PHP is 5.5.9.

- We have 2 media servers set which run on the same server with different domains.

- We don't minify the HTML. According to our tests, if does not help in terms of performance but sometimes results in (smaller) errors on some browsers.

 

Thanks for sharing you experience revving up 1.5.  As a prestanerd and you run 1.5, you might like some one of my old works, once 1.6 eanbled .js bottom it become less relevant except for 1.5 shops.  JavaPro .  I've always known true performance was done on db side and a great area to signifiicanlty improve individual shops.  It's seldom discussed area.  Again thanks for sharing.

[Scully]

Thanks for sharing you experience revving up 1.5.  As a prestanerd and you run 1.5, you might like some one of my old works, once 1.6 eanbled .js bottom it become less relevant except for 1.5 shops.  JavaPro .  I've always known true performance was done on db side and a great area to signifiicanlty improve individual shops.  It's seldom discussed area.  Again thanks for sharing.

 

Hi El Patron,

 

Indeed we used your java pro module for a while but decided to drop it since we had somtimes problems with rarely used javascript functions which resultet in errors.

But we give it a second try.

 

And coming back to thread starters goal to reduce the Time to First Byte TTFB. In our opinion TTFB is not really a goal. You could fool this time by just sending a small part of the HTTP answer in a first step (for example only headers). Interesting is the time to load the whole HTML page.

And interesting is the time needed for the browser to start render the page. The more CSS and expecially Javascript are included, the longer it takes.

[Scully]

Hi El Patron,

 

We again tested the Java Pro Module and did some changes in the settings.

 

It works. However we noticed that the javascript is loaded from the main domain instead of one of the media servers.

Can you check that out? The original way to check for this settings is in ../classes/tools.php.

 

Sorry - my fault: We forgot to enable CCC for javascript which is necessary in order to use CDN / Media Servers.

 

Regards, Scully

 

BTW: We also tested php7 today. It uses between 20 to 40% less memory and executes faster by appx. 20% in average. Cool!

Edited October 9, 2016 by Scully (see edit history)

[xinterceptorx]

@JPresta.com does your Caching module support Prestashop 1.7.x? If yes, I would like to try (30 days) and then most likely purchase a licence

[PrestaHeroes USA]

For the ultimate speed freaks we released last month Google AMP Prestashop Module

[JPresta.com]

@JPresta.com does your Caching module support Prestashop 1.7.x? If yes, I would like to try (30 days) and then most likely purchase a licence

I will release v4 soon (march 2017) with new features and compatibility with PS1.7. There is no trial version but module is satisfied or refunded you have 30 days to test it.

[xinterceptorx]

 

 

I will release v4 soon (march 2017) with new features and compatibility with PS1.7. There is no trial version but module is satisfied or refunded you have 30 days to test it.

 

If I buy a licence now, will I get a free upgrade for the 1.7.x version (v4) that you planning to release this month? I would like to use it on the current 1.6.x shop and then on the 1.7.x shop in a few weeks...

[JPresta.com]

If I buy a licence now, will I get a free upgrade for the 1.7.x version (v4) that you planning to release this month? I would like to use it on the current 1.6.x shop and then on the 1.7.x shop in a few weeks...

Yes you will.

[PrestaHeroes USA]

you all do understand AMP yes?  and  Google mobile first?

[JPresta.com]

you all do understand AMP yes?  and  Google mobile first?

Yes but why is it a module and not a theme?

[PrestaHeroes USA]

Yes but why is it a module and not a theme?

 

if I told you I would have to kill you.  I'm not big static cache guy....so it's best I don't ask to many questions about that if you know what I mean.  Did you want a job with us to help us design an AMP solution?  Maybe we did it wrong?  lol

Edited March 7, 2017 by El Patron (see edit history)

[JPresta.com]

if I told you I would have to kill you.  I'm not big static cache guy....so it's best I don't ask to many questions about that if you know what I mean.  Did you want a job with us to help us design an AMP solution?  Maybe we did it wrong?  lol

I did not dive into AMP yet but it's on my TODOs stack

[BoKr]

2020 November

My hosting uses a virtual manages host. I have ssh access but luckily the host hast preinstalled pagespeed_mod.

question:

is pagespeed_mod better (faster, less ressources, or what so ever) than CCC in terms of packing and merging html, css, js?