Jump to content
Hide-Nene

Trouble uploading product csv files in large quantity

Recommended Posts

I have 1.5.4 store running at SiteGround and have upgraded to their Cloud 1 hosting.

http://www.siteground.com/cloud-hosting.htm

 

The reason for the upgrade from a shared hosting was the fact that I was not allowed to change timeout setting.

 

However, I still have trouble uploading product csv file in large quantity. I would like to upload at least a few thousands products at once. I have read it can be corrected by changing some setting in PHP.ini file. Do you have any recommendation for the setting? or Is there anything else I should be paying attention to?

 

Any advice will be appreciated!

Share this post


Link to post
Share on other sites

The main thing you want to adjust is the timeout limit and the memory limit in the php.ini. You can set the timeout limit to as long as you want 10 or 15 minutes should be enough, but depending on your image size it could take longer. One thing to keep in mind is say you set your php to time out in 15 minutes. A browser window will only let you load the csv page for 10 minutes, so after 10 minutes you will either get a 404 error or a 500 server error. This affects nothing. Go to the product section of your shop and see if all of the products are uploaded. If they are not, don't worry, wait a couple of minutes and refresh the page and see if any new products have been loaded. Even though your browser window times out, the import script will keep running until either the php execution times out, an error is reached in the file, or the import is finished.

  • Like 2

Share this post


Link to post
Share on other sites

There's one more thing to it - you can skip lines in the CSV import, so if the php scripts runs out of time, you can simply go to BackOffice-Catalogue-Products and in case the ID of the products is rising from 1 to X, you can click the sort button, from decreasing ID. Now you should see the last product that was imported, say #1000. Now, you can skip 1000 lines so you'll continue where the script stopped last time.

Btw a VPS server has usually also some limits, so you probably won't be able to increase the timeout indefinetely..

  • Like 1

Share this post


Link to post
Share on other sites

VPS's are containers that don't have limits. You can ssh into the main php.ini and adjust it there. The only real limit is processor share time. I have imported over 15k products in a go on a vps before.

  • Like 2

Share this post


Link to post
Share on other sites

Thank you all of you for your advise.

 

I will explain my situation more in detail as follows and apprecaite your further advice on the trouble.

 

My Prestashop 1.5.4 is running on SiteGround's Cloud 1 (VPS hosting).

 

Siteground Cloud 1 hosting:

CPU: 2x2.0GHz

RAM: 1024MB DDR3

Disk Space:40GB

Bandwidth: 1TB

 

Setting for php.ini

 

memory_limit = 128M

max_execution_time=1800;

max_input_time=1800

upload_max_filesize=2M;

post_max_size=8M;

max_input_vars=2000

suhosin.get.max_vars: NOT INSTALLED

suhosin.post.max_vars: NOT INSTALLED

 

As we receive a blank page (not uploaded) even with 100 products, we overrided the setting as follows;

 

Overrided Setting of AdminImportController.php

@ini_set('memory_limit', '1024M');

@ini_set('max_execution_time', 0);

@ini_set('max_input_time', -1);

@ini_set('max_input_vars', 20000);

 

Fields of uploaded csv fileCSV

 

ID

Active (0/1)

Name *

Categories (x,y,z...)

Price tax excluded

Reference #

Manufacturer

EAN13

Weight

Quantity

Description

Image URLs (x,y,z...) * one picture per product, 100KB max. in size per picture

Feature(Name:Value:Position)

 

It takes approx. 40min. to upload 1000 products successfully.

When uploaded the same file with NO IMAGES, it only takes approx. 40sec.

 

When tried more products with images (say, 2000), we will receive Not Found Page.

It sometimes stops after 5min. Sometimes after 30min.

 

I have more than 100,000 products to upload and need to upload files with at least 10K products at once

Is there anyway to improve the performance of uploading the files?

 

Thank you for your attention and your advice!

Share this post


Link to post
Share on other sites

Hmm are the images on the same server or on a different one? If you have the files made already, I think I could import them into a shop in 1 day, two days max.

 

 

The issue might be the amount of processor share that your site is getting on the server it is on. It takes quite a bit of processing power to import a lot of products.

Share this post


Link to post
Share on other sites

The images are on a different server. I appreciate your offer, but there are a lot more files to uploads, so I need to find out what setting need to be changed. Do you have any ideas for the solution?

 

Thank you.

Share this post


Link to post
Share on other sites

I would move the files to the same server to save a bit a time. The only thing you can really do to make things move faster is to move to a faster server. With that many products you should be using a dedicated server.

Share this post


Link to post
Share on other sites

Thank you for your advice, Dh42

I will try putting images in the same server and try.

If it does not solve the problem, I would consider upgrading the server.

Share this post


Link to post
Share on other sites

Hello! I am running into the same requirement as Hide-Nene, could I get some personalized support and get some professional from your team to help me out?

Share this post


Link to post
Share on other sites

×
×
  • Create New...

Important Information

Cookies ensure the smooth running of our services. Using these, you accept the use of cookies. Learn More