404 error when CSV uploading

[JoelWebsites]

404 error when CSV uploading ..I am experiencing issue when uploading CSV products ..it the process shows me a 404 error after some minutes...How do I avoid this issue ? and upload the products without any issues?..I also get 404 errors when regenerating the image thumbnails sometimes.

[Guest]

This will be time out issues. Try this as a starting point

 

http://dh42.com/blog/speed-prestashop-php-ini/

[JoelWebsites]

Thanks for the answer ..however this did not solve the issue..It happens only when the CSV file has product image links..

[PascalVG]

Hi Joel,

 

Can you check (in both cases) the time after which you get the error page? Is that sometimes after a few seconds and sometimes much later in the process, or always say after say 30 seconds, or 60 seconds etc.? If the latter, the timeout would still be a reasonable explanation...

 

My 2 cents,

pascal

[Guest]

Actually drawing down issues could well be the issue as that part seems to take a lot longer. Try with just one product. If that works then it is likely a time out issue. If not then we look further

[JoelWebsites]

I dont get it when i upload 1 product... I get loggout of my prestashop admin panel..it tells to loggin again...i have set the maximum exeution time to 1000.. I feel its showing the 404 error because its loggin me out of the admin back office..i have also disabled dynamic ip in general section in preferences general section.. But i still get logged out at times..

[Guest]

OK, try switching on Debug mode and see if you get any errors

1. open file: config/defines.inc.php  (using FTP or hosting control panel)
2. change false-->true:

define('_PS_MODE_DEV_', false);  change to:

define('_PS_MODE_DEV_', true);

[JoelWebsites]

I am getting the 404 error exactly after 15 minutes and I have out error reporting ON too..

[Guest]

Take this one step at a time. Do only this then report the result

 

1) Switch on debug as shown above

2) Create CSV file with only 1 product, but without the image link

3) Import the CSV file

 

Stop now. If that works then continue below, if it does not work then come back here and report on error messages (also check your log files)?

 

Assuming the above does work then continue

 

4) Create CSV file with 1 product and this time include the image link

5) Import CSV file

 

Stop now. Does that work? if yes then your problem is definitely a server issue. Either timeout or memory limits

If it does not work then report back here with the error messages (again check error logs)

 

When you reply try to be very logical in the reply giving us details

[JoelWebsites]

I have tried with a single product it worked properly , I did not get any error messages..I have also updated some parameters 

maximum_execution_time 10000,

max_input_vars 1000000,

memory_limit 4048M ,

post_max_size 70M,

upload_max_filesize 100M

 

but it was of no use...when i upload many products then i get 404 error after 15 minutes exactly.Are there any other parameters that I could change to try to solve this issue?..

 

Or is there a 15 min limit somewhere in prestashop?

Edited April 11, 2015 by JoelWebsites (see edit history)

[PascalVG]

Hi Joel,

 

Can you check if you have a file  /usr/local/etc/wgetrc    (or maybe somewhere else than usr/local/   )

 

And check if you have the value 900 in that file, probably something like this:

 

timeout = 900

 

My 2 cents,

pascal.

[Guest]

Or check with your server host. Even though you can modify php settings, they may be over riding them

 

Have you checked your actual settings with a phpinfo file?

http://www.inmotionhosting.com/support/website/php/create-phpinfo-page-to-see-php-settings

[JoelWebsites]

I have checked the phpinfo ...the settings have changed properly..i even tried on mozilla firefox but i get 404 after exactly 15 mins. I have also checked the wgetrc there was no parameter like timeout = 900,So I added timeout =1800 but still this did not solve the issue...Please note that I  get logged out of the prestashop admin panel as well when I get the 404 error page.

[PascalVG]

Think I found it!

 

 

In file classes/controller/AdminController.php, there is a function init file which has a time out of a (hard coded!!! Shame, Prestaguys!!!) 900 in it:

 

 

public function init()

{

  // Has to be removed for the next Prestashop version

  global $currentIndex;

 

  parent::init();

 

  if (Tools::getValue('ajax'))

    $this->ajax = '1';

 

  /* Server Params */

 

  $this->context->link = new Link($protocol_link, $protocol_content);

 

  if (isset($_GET['logout']))

    $this->context->employee->logout();

 

  if (isset(Context::getContext()->cookie->last_activity))

  {

    if ($this->context->cookie->last_activity + 900 < time())

      $this->context->employee->logout();

    else

      $this->context->cookie->last_activity = time();

  }

...

 

I expect this to be the culprit here. Change to something higher and try again (or try a lower amount and see if it indeed does what you normally see after 15 minutes

 

 

Hope this is it, let me know.

pascal

[JoelWebsites]

I have changed the value to a greater number that is 100000000000000 but it did not solve the issue..

I even tried commenting out the code in the file classes/Cookie.php the logout function..but this did not help resolve the issue...Is there any other way I could try to fix this issue?

 

public function logout()
{
/*$this->_content = array();
$this->_setcookie();
unset($_COOKIE[$this->_name]);
$this->_modified = true;*/
}

Edited April 14, 2015 by JoelWebsites (see edit history)

[PascalVG]

can you try one more time, using a little smaller number, like just  9000?

 

 

Otherwise change code to:

  if (isset(Context::getContext()->cookie->last_activity))

  {

//    if ($this->context->cookie->last_activity + 9000 < time())

//      $this->context->employee->logout();

//    else

      $this->context->cookie->last_activity = time();

  }

 

 

See if that helps.

[JoelWebsites]

I tried again by commenting out the code but it did not solve the issue....does the token on the URL have some time limit?

 

 

if (isset($_GET['logout']))
//$this->context->employee->logout();


if (isset(Context::getContext()->cookie->last_activity))
{
//if ($this->context->cookie->last_activity + 9000 < time())
// $this->context->employee->logout();
//else
$this->context->cookie->last_activity = time();
}

[PascalVG]

Hi Joel,

 

Did you clear your cookies before trying again?

 

Try to turn off all caching, ccc optimisations etc and see if that helps.

 

Cannot find that much else that smells like it can be of any influence:

 

Maybe file:

    /config/config.inc.php:

 

which has this line:

    $cookie_lifetime = time() + (max($cookie_lifetime, 1) * 3600);

 

    if (defined('_PS_ADMIN_DIR_'))

        $cookie = new Cookie('psAdmin', '', $cookie_lifetime);

 

 

But this seems to be an hour at the minimum..

 

 

or file:

    classes/Connection.php

in function:

    public static function setNewConnection($cookie)

it has:

    AND `date_add` > \''.pSQL(date('Y-m-d H:i:00', time() - 1800)).'\'

 

But I don't see any direct 'connection' (pun intended)

 

 

there is an expire time in classes/Cookie.php:

    $this->_expire = is_null($expire) ? time() + 1728000 : (int)$expire;

 

But $expire is here the $cookie_lifetime as defined in config/config.inc.php (see above, where the new $cookie is created)

 

 

I really think it's in the AdminController.php. Maybe it's not compiling for some reason. Do you have any additional server side caching system as well?

 

pascal.

[PascalVG]

Maybe try to add some nonsense syntax line to the AdminController File and see if it complains. Then at least we know that it sees and compiles the file...

[satkauskas]

I have same issue.

 

What about adding some php lines into admincontroller file? e.g. stats by second? I mean - a logging with output into browsers window, "how the import is doing now"

 

What about finding out the statistics from phpmyadmin side? e.g. on my side:

mysql : Aborted clientsDokumentacija 33,6 k The number of connections that were aborted because the client died without closing the connection properly.,  

Aborted connects 3,1 k

 

It seems that mysql has a timeout on my server?

 

my: 

PrestaShop Version 1.6.1.3 PHP Version 5.6.16 (OK) MySQL Version 5.5.34-cll-lve (Consider upgrading) Memory Limit 6024M Max Execution Time 300s Smarty Cache disabled Smarty Compilation auto Edited December 29, 2015 by satkauskas (see edit history)

[satkauskas]

Even these are not helping for config/config.inc.php

 

/* Improve PHP configuration to prevent issues */

 

ini_set('default_charset', 'utf-8');

ini_set('magic_quotes_runtime', 0);

ini_set('magic_quotes_sybase', 0);

 

ini_set('max_execution_time','15000');

ini_set('max_input_time','15000');

ini_set('default_socket_timeout','15000');

 

ini_set('upload_max_filesize', '100M');

ini_set('memory_limit', '4000M');

ini_set('query_cache_size', '100M');

ini_set('post_max_size','200M');

 

ini_set('session.gc_maxlifetime', 15000);

ini_set('mysql.connect_timeout', 15000);

ini_set('max_input_vars', 200000);

ini_set('expect.timeout', '15000');

ini_set('default_socket_timeout', '15000');

 

ini_set('max_allowed_packet','250M');

 

ini_set('suhosin.request.max_vars', 100000);

ini_set('suhosin.post.max_vars', 100000);

 

 

and these for controllers/admin/AdminImportController.php

 

@ini_set('memory_limit', '6024M');

@ini_set('session.gc_maxlifetime', 300);

@ini_set('max_execution_time', 300);

@ini_set('mysql.connect_timeout', 300);

@ini_set('max_input_time', 300);

@ini_set('max_input_vars', 100000);

@ini_set('expect.timeout', '300');

@ini_set('default_socket_timeout', '300');

 

 

these two settings are enabled, I can see on "dev mode true"

[satkauskas]

servers'   MYSQL settings;

 

my.cnf :

max_connections = 200
interactive_timeout = 60
wait_timeout = 60
connect_timeout = 10
max_user_connections = 50
log_slow_queries = /var/log/mysql/mysql.slow.log
#log-queries-not-using-indexes
long_query_time = 2
tmp_table_size = 1024M
max_heap_table_size = 1024M
join_buffer_size = 64M
thread_cache_size = 8
table_cache = 4M
query-cache-type = 1
query-cache-size = 2048M
innodb_buffer_pool_size = 1024M
thread_concurrency = 8
max_allowed_packet=268435456
open_files_limit = 1000000
key_buffer_size = 256M
sort_buffer_size = 8M
read_buffer_size = 8M
read_rnd_buffer_size = 16M
table_cache = 4096

innodb_additional_mem_pool_size = 64M
innodb_log_buffer_size = 64M
innodb_flush_method=O_DIRECT

[satkauskas]

some updates:

 

>>  MySQLTuner 1.6.1 - Major Hayden <[email protected]>
>>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
>>  Run with '--help' for additional options and output filtering
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.34-cll-lve
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[--] Data in MyISAM tables: 3G (Tables: 1861)
[--] Data in InnoDB tables: 2G (Tables: 12821)
[!!] Total fragmented tables: 485

-------- Performance Metrics -------------------------------------------------
[--] Up for: 14d 22h 23m 17s (51M q [39.707 qps], 805K conn, TX: 114B, RX: 15B)
[--] Reads / Writes: 81% / 19%
[--] Binary logging is disabled
[--] Total buffers: 4.4G global + 96.2M per thread (200 max threads)
[!!] Maximum reached memory usage: 19.1G (255.64% of installed RAM)
[!!] Maximum possible memory usage: 23.2G (309.65% of installed RAM)
[OK] Slow queries: 0% (10K/51M)
[OK] Highest usage of available connections: 78% (157/200)
[OK] Aborted connections: 0.39%  (3137/805379)
[OK] Query cache efficiency: 67.1% (28M cached / 41M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 2M sorts)
[!!] Joins performed without indexes: 6033
[!!] Temporary tables created on disk: 50% (1M on disk / 3M total)
[OK] Thread cache hit rate: 99% (4K created / 805K connections)
[!!] Table cache hit rate: 0% (4K open / 1M opened)
[OK] Open file limit used: 0% (1K/1M)
[OK] Table locks acquired immediately: 99% (17M immediate / 17M locks)

-------- MyISAM Metrics -----------------------------------------------------
[!!] Key buffer used: 19.6% (52M used / 268M cache)
[OK] Key buffer size / total MyISAM indexes: 256.0M/639.6M
[OK] Read Key buffer hit rate: 99.6% (529M cached / 2M reads)
[!!] Write Key buffer hit rate: 89.9% (26M cached / 2M writes)

-------- InnoDB Metrics -----------------------------------------------------
[--] InnoDB is enabled.
[!!] InnoDB buffer pool / data size: 1.0G/2.4G
[OK] InnoDB buffer pool instances: 1
[OK] InnoDB Used buffer: 100.00% (65535 used/ 65535 total)
[OK] InnoDB Read buffer efficiency: 98.76% (647056232 hits/ 655172867 total)
[!!] InnoDB Write buffer efficiency: 0.00% (0 hits/ 1 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1134826 writes)

-------- AriaDB Metrics -----------------------------------------------------
[--] AriaDB is disabled.

-------- Replication Metrics -------------------------------------------------
[--] No replication slave(s) for this server.
[--] This is a standalone server..

-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    Restrict Host for user@% to user@SpecificDNSorIp
    Reduce your overall MySQL memory footprint for system stability
    Adjust your join queries to always utilize indexes
    Temporary table size is already large - reduce result set size
    Reduce your SELECT DISTINCT queries without LIMIT clauses
    Increase table_open_cache gradually to avoid file descriptor limits
    Read this before increasing table_open_cache over 64: http://bit.ly/1mi7c4C
    Beware that open_files_limit (1000000) variable
    should be greater than table_open_cache ( 4096)
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    join_buffer_size (> 64.0M, or always use indexes with joins)
    table_open_cache (> 4096)
    innodb_buffer_pool_size (>= 2G) if possible.
 

 

Made my.cnf changes according to recommendations:
[mysqld]
user = mysql
innodb_file_per_table = 1
symbolic-links = 0
datadir = "/var/lib/mysql"
socket = "/var/lib/mysql/mysql.sock"
tmpdir = "/tmp"

collation_server = utf8_unicode_ci
character_set_server = utf8

max_connections = 200
interactive_timeout = 60
wait_timeout = 60
connect_timeout = 10
max_user_connections = 50
log_slow_queries = /var/log/mysql/mysql.slow.log
#log-queries-not-using-indexes
long_query_time = 2
tmp_table_size = 128M
max_heap_table_size = 128M
join_buffer_size = 64M
thread_cache_size = 8
table_cache = 4M
query-cache-type = 1
query-cache-size = 128M
innodb_buffer_pool_size = 3072M
thread_concurrency = 8
max_allowed_packet=268435456
open_files_limit = 1000000
key_buffer_size = 96M
sort_buffer_size = 8M
read_buffer_size = 8M
read_rnd_buffer_size = 16M
table_cache = 4096

innodb_additional_mem_pool_size = 64M
innodb_log_buffer_size = 64M
innodb_flush_method=O_DIRECT

[satkauskas]

Interesting, if SEO urls are enabled I get 404 error. If SEO urls are disabled i get 500 error.

Regenerate using SEO URLs

1. Log into your PrestaShop admin panel.

2. Navigate to Preferences -> SEO and URLs.

3. Set Friendly URL option to No and click Save button:

4. Enable Friendly URL back and save the changes.

[satkauskas]

solution is found now drop a message here if needed.

A description is here: https://www.prestashop.com/forums/topic/596062-csv-import-time-out-solution-prestashop-16-solved/

Edited February 27, 2017 by satkauskas (see edit history)

[Artunj]

Hi

 

I have the same issue, please help how to solve it?

[octb]

solution is found now drop a message here if needed.

 

Can you please share your solution? Having the same problem. Thanks, you`re awesome!

Edited February 1, 2016 by octb (see edit history)

[marcelderuig]

For me i found the solution.

 

I found that settings->SEU/URLS->Friendly URLS switched to on is giving 404-error.

Switching it off gives me an error 500.

Looking in the logfiles it shows:

mod_fcgid: read data timeout in 600 seconds

 

Found link: http://expressionengine.stackexchange.com/questions/7467/mod-fcgid-read-data-timeout-in-45-seconds-premature-end-of-script-headers

 

Quote: That error is specific to mod_fcgid, not PHP itself. If you're using the current version of mod_fcgid then you will need to set the FcgidIOTimeout directive (ref), otherwise you'll need the old directive, IPCCommTimeout

 

Changed this timeout in /etc/apache2/sites-available in the according .vhost-config, belonging to your site.

 

After the import i changed it back.

[BricksDirect]

For me i found the solution.

 

I found that settings->SEU/URLS->Friendly URLS switched to on is giving 404-error.

Switching it off gives me an error 500.

Looking in the logfiles it shows:

mod_fcgid: read data timeout in 600 seconds

 

Found link: http://expressionengine.stackexchange.com/questions/7467/mod-fcgid-read-data-timeout-in-45-seconds-premature-end-of-script-headers

 

Quote: That error is specific to mod_fcgid, not PHP itself. If you're using the current version of mod_fcgid then you will need to set the FcgidIOTimeout directive (ref), otherwise you'll need the old directive, IPCCommTimeout

 

Changed this timeout in /etc/apache2/sites-available in the according .vhost-config, belonging to your site.

 

After the import i changed it back.

Dear Marcel,

 

What did you do exactly to solve this problem?

I don't understand.

 

Can you let me know?

 

Thank you.

Best regards,

Niels

[satkauskas]

Please take a look here ( adding some extra option to increase timeouts and memory ) - https://www.prestashop.com/forums/topic/596062-csv-import-time-out-solution-prestashop-16-solved/

[BricksDirect]

Sorry but it doesn't work for me.