Using long running Cron Jobs with Prestashop and Cloudflare CDN

[cyanoxide]

So I've got Cloudflare and Prestashop running together to improve load times however a side effect of the CDN is that my cronjobs can only run for a maximum of 90 seconds. Any longer and Cloudflare will send out a 524 error and the cronjob will not be properly launched.

According to Cloudflare there are two ways to get around this problem. I can either reduce the size of the cronjob process so it fits within the 90 second window (which is not an option) or I can run the cronjobs on a separate subdomain that Cloudflare has no effect on. The problem with this option however is that Prestashop has something built into it so that regardless of what subdomain you use to visit the site it simply redirects to the main domain.

Does anyone have experience with this kind of issue and if so what are the best methods of getting around the problem. Thanks!

 

Additional notes: In my research into this I've found two possible solutions but I don't know how I would go about doing either of them so if you have any ideas it would be greatly appreciated.

The first being that I could separate the cronjobs up into several smaller jobs that run together however the cronjobs I wish to use all come from different modules so I'm assuming this would involve extensive modding to the module code to get working. If there is a simple way of doing this I'd be very interested but I'm assuming that's not the case.

The other alternative is to get the cronjobs to somehow run through the back office as that area of the website is not affected by the subdomain redirect limitation that the front end of the site has. I'm not sure how I would go about doing this however so if anyone could help me with that it would be greatly appreciated.

Both of these solutions seem to involve some modding to the module and ideally I'd prefer a solution that didn't rely on that so I'm free to add new jobs in the future without having to worry about rebuilding them prior so please let me know if you know of any solutions that will allow this.

[DataKick]

That is an interesting problem.

 

It's a pity that prestashop forces redirection to the main domain, as separate subdomain that bypasses cloudflare proxy would definitely solve this problem. It shouldn't be hard to modify prestashop core to disable this auto-redirect though, so this might be the viable solution.

 

I'm afraid every other solution would require changes to every module's cron implementation. One possibility is to add a small code that instructs php server not to end cron process when connection is closed by client (or in this case proxy server). Or proactively close the connection from server side, but continue to run the cron process. Something like this:

  // close connection and continue on backend only
  ignore_user_abort(true);
  ob_start();
  echo 'cron: closing connection';
  header('Connection: close');
  header('Content-Length: '.ob_get_length());
  ob_end_flush();
  ob_flush();
  flush();
  if (function_exists('fastcgi_finish_request'))    
    fastcgi_finish_request(); 

But as I said, you would have to modify every cron php file in all the modules

[cyanoxide]

That is an interesting problem.

 

It's a pity that prestashop forces redirection to the main domain, as separate subdomain that bypasses cloudflare proxy would definitely solve this problem. It shouldn't be hard to modify prestashop core to disable this auto-redirect though, so this might be the viable solution.

 

I'm afraid every other solution would require changes to every module's cron implementation. One possibility is to add a small code that instructs php server not to end cron process when connection is closed by client (or in this case proxy server). Or proactively close the connection from server side, but continue to run the cron process. Something like this:

  // close connection and continue on backend only
  ignore_user_abort(true);
  ob_start();
  echo 'cron: closing connection';
  header('Connection: close');
  header('Content-Length: '.ob_get_length());
  ob_end_flush();
  ob_flush();
  flush();
  if (function_exists('fastcgi_finish_request'))    
    fastcgi_finish_request(); 

But as I said, you would have to modify every cron php file in all the modules

 

Thanks for your response, if there is a way to disable the redirect or at least exclude a certain subdomain from being a part of it then that would be the perfect solution. I'm not sure how I would go about doing that though unfortunately.

[DataKick]

Thanks for your response, if there is a way to disable the redirect or at least exclude a certain subdomain from being a part of it then that would be the perfect solution. I'm not sure how I would go about doing that though unfortunately.

 

I did a quick test on my local installation (1.6.1.7) - I've edited line 370 of classes/shop/Shop.php

 

        if ((!$id_shop && defined('_PS_ADMIN_DIR_')) || Tools::isPHPCLI() || in_array($http_host, $all_media) || $http_host == 'exemption.myshop.com') {

Note that the redirection still occurs if you try to open 'normal' pages such as product list or categories - there is one more check in front controller that perform redirection. 

 

But cron urls are not redirected because they don't use controllers (usually)

[cyanoxide]

Thank you for your help on this matter, unfortunately your solution didn't resolve the problem but I think I'm making progress with it. When ever I run the script not I'm hit with a 504 error instead of the 524 error I was hitting before. It's worth noting that the whole process runs with the subdomain in the url but when it hits the error it converts to www. which could be common behavior or it could indicated something else.

 

At first I figure nothing had changed but I timed the process and it stops at exactly 60 seconds (I didn't time it before but I was under the impression that it was 90 seconds as that's cloudflare's limit) which made me think that maybe there was a configuration file like php.ini that had something set at 60. I went in there and changed the following parameters:

 

mysql.connect_timeout

default_socket_timeout

max_input_vars

max_execution_time

max_input_time

 

I basically added a 0 to the end of each of the values to see if they would have any effect. I checked my info.php file to confirm the changes had been made but it didn't change anything. The process still times out and gives me the 504 error. I'm not really sure where to go from here.

[DataKick]

504 error code suggests that there is some proxy server/gateway in front of your web server that causes this timeout. The question is where is this proxy located. It can be on your webserver (maybe nginx reverse proxy?), inside the network infrastructure of your hosting provider, or even in your very own network (bad settings of your home router, or your ISP provider,...)

 

Try to contact your hosting provider and inquire whether they have some sort of proxy server in place...

[cyanoxide]

504 error code suggests that there is some proxy server/gateway in front of your web server that causes this timeout. The question is where is this proxy located. It can be on your webserver (maybe nginx reverse proxy?), inside the network infrastructure of your hosting provider, or even in your very own network (bad settings of your home router, or your ISP provider,...)

 

Try to contact your hosting provider and inquire whether they have some sort of proxy server in place...

 

I've spoken to my provided and managed to resolve the timeout issue but it seems I'm back to hitting the 524 error and it redirecting from the sub domain to www. so it would seem the above code amendment to shop.php isn't working as intended.

[DataKick]

hmm, it must hit the redirection in FrontController then.

 

Try to modify classes/controller/FrontController.php, method canonicalRedirection, lines ~811. Add this code (don't forget to use your subdomain)

 

        if (Tools::getHttpHost() == 'exemption.myshop.com')
          return;

That's the last redirection I could find in the codebase. 

[cyanoxide]

 

hmm, it must hit the redirection in FrontController then.

 

Try to modify classes/controller/FrontController.php, method canonicalRedirection, lines ~811. Add this code (don't forget to use your subdomain)

 

        if (Tools::getHttpHost() == 'exemption.myshop.com')
          return;

That's the last redirection I could find in the codebase. 

 

 

Would you mind posting the full function so I can see exactly where you've placed it within canonicalRedirection? I put it at the top and it still had no effect. I appreciate your help on this, thank you.

[DataKick]

Would you mind posting the full function so I can see exactly where you've placed it within canonicalRedirection? I put it at the top and it still had no effect. I appreciate your help on this, thank you.

 

Hi

 

top of the function is the best place to put this new code.

 

I tried it once more on my production site (I'm using cloudflare as well) and found out that there's one more redirection in case you are using secure connection - SSL. I'm not using SSL in my local environment so I miss this redirection before.

 

If you are using SSL, try to add the same code as you added to canonicalRedirection function to the top of the sslRedirection function in FrontController (this function is located just above canonicalRedirection function)

 

You can see that it's working correctly on my demo account - for example this url:

 

https://demo.getdatakick.com/en/tshirts/1-faded-short-sleeves-tshirt.html

 

vs

 

http://direct.getdatakick.com/en/tshirts/1-faded-short-sleeves-tshirt.html

 

Note: you need to access the new subdomain using http, not https, protocol

Edited January 19, 2017 by DataKick (see edit history)

[cyanoxide]

Hi

 

top of the function is the best place to put this new code.

 

I tried it once more on my production site (I'm using cloudflare as well) and found out that there's one more redirection in case you are using secure connection - SSL. I'm not using SSL in my local environment so I miss this redirection before.

 

If you are using SSL, try to add the same code as you added to canonicalRedirection function to the top of the sslRedirection function in FrontController (this function is located just above canonicalRedirection function)

 

You can see that it's working correctly on my demo account - for example this url:

 

https://demo.getdatakick.com/en/tshirts/1-faded-short-sleeves-tshirt.html

 

vs

 

http://direct.getdatakick.com/en/tshirts/1-faded-short-sleeves-tshirt.html

 

Note: you need to access the new subdomain using http, not https, protocol

 

 

That seems to have done the trick!

 

I added the code into the SSL redirect section and gave it a go. It ran a lot longer, I got another timeout after 5 minutes but that wasn't a problem, I just increased the time out setting in HA Proxy from 5m to 10m and it worked perfectly.

 

Thank you for all your help on this, it is very much appreciated.

[DataKick]

That seems to have done the trick!

 

I added the code into the SSL redirect section and gave it a go. It ran a lot longer, I got another timeout after 5 minutes but that wasn't a problem, I just increased the time out setting in HA Proxy from 5m to 10m and it worked perfectly.

 

Thank you for all your help on this, it is very much appreciated.

 

Glad I could help. It's good to know about this cloudflare restriction - I myself haven't reached this timeout yet, but it's better to be prepared.

[cyanoxide]

Glad I could help. It's good to know about this cloudflare restriction - I myself haven't reached this timeout yet, but it's better to be prepared.

Yeah, I was quite surprised at the amount of steps than need to be taken to get it working. Granted Prestashop is partly to blame but you'd think there would be a much easier process. I'm also quite shocked that there's no documentation on this issue anywhere. I hope this thread will help others with the same problem in the future.

[hartvigmorten]

For others: If you use Nginx reverse proxy, you get the 60 second limit as default. (Gateway timeout)

[Skafander]

thanks for this tip. I did such modification, and my script is unfortunately still redirected to the domain. I'm using Prestashop 1.6. This script have 3 steps, additionally our store uses token link security. Does anybody know where are additional changes necessary?

What kind of redirection subdomain suppose have, 301, 302 or Frame?

[JuanjoSC]

Hi,

Although it is a very old topic, looking for a solution to the problem I have found it.
So I add the solution without modifying anything in the prestashop.

You have to use curl in the cron with these parameters:

curl --resolve example.org:443:1.2.3.4 https://example.org/path?and=query -sL

For more information:

https://tomssl.com/bypassing-cloudflare-for-long-running-tasks-without-exposing-your-ip-address/

 

Thanks to @joseantgv