Jump to content

Prestashop fail to import 1000+ product CSV file.


Recommended Posts

Ok here is the issue.

 

I prepared CSV file with my products. It has around 1900 entries.

In the middle of the import process, something happens and it redirect me to some unknown page on the front end. No warnings, errors noting...

 

When I go back to my back office I find that around 100+ products are imported and rest is not.

I tried this several times, sometimes I get 170 products, some times 220 etc...

 

I contacted my hosting provided and they confirmed that all is good at their end.

 

What is wrong?

Link to comment
Share on other sites

This has been an issue on my end as well. I've been mucking around with the php.ini configuration as well as httpd.conf trying to get it to stick with it. I may end up contacting the hosting company (GoDaddy) to see if they can figure out what's wrong.

Link to comment
Share on other sites

Hi,

 

Thanks for sharing. I host at HostGator.com and I already spoke with them regarding this issue thinking that problem might be in some server or php configuration. After they tried few things and I repeated the import procedure few times with 1000+ but also with less then 1000 products I always get same results.

 

So, after I press import button process begins and it takes usually around 5 minutes.

After that I'm transferred to page on my front end that doesn't exist instead to be taken into back end and showed errors and warnings.

 

I would like to hear from some Presta developer, because they have been kind to answer my pre presta questions (one of them was importing large numbers of products using CSV import feature) and now when obviously this doesn't seem to work ok, there are no presta experts nor core developers to post in this topic. I wonder why but I do have a guess...

Link to comment
Share on other sites

Update:

 

I was deleting big number of items in the back office at once and I got the same error in one occasion.

 

Can someone from the PS please come forward and give us some heads up here???

Link to comment
Share on other sites

The most frustrating part is: this is technically a solved problem. Here a guy figures out how to get 1.5 million rows from a csv into his SQL table (just like prestashop, though more columns involved) and make a change on the backend to do it row-by-row. No time outs, no crashes, and he inserts 1.5 million rows in just 10 minutes.

 

If we could get one active PrestaShop coder in here to add a few lines and push out an update, it would make dozens of users very happy. Isn't that the point?

  • Like 1
Link to comment
Share on other sites

Hello all, could you attach your CSV file, Alexanderm920, I'm not sure if we can help too much, GoDaddy has shown to have very strange PrestaShop issues for years, including CSV.

 

Attach your CSV files and I'll do my best to let you know if it's a formatting error or a server issue.

 

 

Here is the csv file. I'd figure if there was a format issue it wouldn't start importing at all. I managed the import settings to identify the columns and skip the first row, so that shouldn't present an issue.

 

soft.zip

Link to comment
Share on other sites

I've been trying to break the file into smaller and smaller portions and noticed a new problem. At 50 or 25 rows per CSV, I get multiple entries in the final list of products. I get two entries for each one entry in my CSV, meaning 100 or 50 entries after everything is said and done.

 

I have no idea what is causing this, and I haven't seen anyone else reporting similar issues. I'm right on the edge of giving up on PrestaShop, though every other feature works as intended the importing simply doesn't.

Link to comment
Share on other sites

Hello Mr. Benjamin,

 

Here is my CSV file:

 

ArtikliImport2405 IZMENJENO.zip

 

 

As you can see it has around 3000 entries.

 

I finally manage to import all product to my website but I needed to break this big CSV file into around 25-30 smaller files where each contained around 100 products in order to do so.

 

I think problem was with my current hosting environment (shared hosting at HostGator.com) that dont allow that much time for scripts to execute so import process was breaking randomly.

 

Problem is not with the breaking it self, as I said before we can blame hosting for that, but the result which is page on my front office that doesn't exist at all. I wasn't redirected back to back office and displayed with some sort of error msg from which I could read what is the actual problem and try to solve it, instead I was completely blind.

 

When I finally figured this one out, I faced problems with the length of some fields like meta_description, meta_title and so on.

 

Is it possible that there is no documentation covering this import process, CSV files and each field that CSV file can contain? Am I mistaking or there is no such document or paper?

 

These are all my observations and experiences using Prestashop.

I would like to make it better while I'm working on it and help community to correct bugs.

 

As we speak I'm waiting for my website to be transferred on to VPS as that was major conclusion in the discussion about what prestashop website should use as hosting platform. I believe that you, Mr. Benjamin, have mentioned VPS as optimal solution for prestashop website in the early beginnings of the website development and life so I have chosen HostGator VPS lvl 3 for now.

 

I hope I made good choice.

 

Thank you in advance

Dean

  • Like 1
Link to comment
Share on other sites

What is happening is your server is timing out on the php execution. When it does, it send you to that 404 page. You need to raise the execution time for you php files by editing your php.ini file if your server allows. I would set it at around 20 -30 minutes during the import and reduce it after that. One problem you might run into is that it will still throw the 404 page after 10 minutes, that is because your browser will only spend 10 minutes loading a page. But if everything i done correctly, you can go to your catalog and refresh it and notice that products are still being added as the script is still running.

 

 

edit:

 

Also, if your sever does not let you change the execution time, just break your csv up into 3 different files and load them one at a time. 1900 products, say do 650 in each file and everything should work fine.

  • Like 2
Link to comment
Share on other sites

Hi,

 

Thank you for taking interest in my case.

After a day or 2 trying to make this work and after chating with my hosting support I got to the same conclusion myself.

 

I split my CSV file but not in 3, but in 25-30 smaller CSV files containing around 100 products each and this way I managed to import all of my products. I believe that these time execution limitations vary from host to host...

 

As I write this my website is transferring to VPS environment and these php execution limitations should be gone.

I also hope that general performance of Presta should be better on VPS.

 

But as I said before, this is only the top of the iceberg... I think documentation is very poor when it comes to presta, especially when it comes to CSV import, SQL manager and other, lets call them, advanced features.

 

I also wrote down an example, that even when I figured this server side problem and solved it, i was receiving some errors regarding numbers of characters in various fields in my CSV file, and that is mostly because there is no document (or I haven't found it yet) explaining all these fields, what they can and cant hold, what type of characters, how long is the string you can input and so on.

 

Do you agree?

Thank you

Dean

  • Like 1
Link to comment
Share on other sites

Dean,

I have been having almost exactly the same experience as you in general. I definitely agree about the explanation and documentation for the advanced features like the csv importer, too.

 

Thanks for helping get this issue out there for everyone to deal with! Also, thanks to the moderator for addressing it. Now I can see if Bluehost can change some settings.

 

Thanks again!

Helen

Link to comment
Share on other sites

Not a problem Helen.

 

I'm sure there are lot of people with a lot of questions out there regarding Prestashop. I'm new with the Prestashop, new with e-commerce software in general, and as in other field there is no perfect solution. If there was such a solution I believe that would be also the only one B).

 

I'm glad that we have exchanged some important information here and I hope that more will come.

 

I will write about my experiences now when Im on the VPS. It might be helpful for those still on shared hosting plans.

 

Dean

Link to comment
Share on other sites

So when I contacted my host, they didn't seem to have in mind what setting or configuration to try changing. They did say there would be a max upload time or file size set somewhere in the server. This seems strange they didn't know what to change since they are the server.

 

Does anyone know what I need to change to increase the file upload maximum size or the time limit?

Link to comment
Share on other sites

seems strange they didn't know what to change since they are the server.

 

I agree, I think that maybe you should call again after reading Dh42 article. It is concerning the php.ini file and variables like max_execution time. BlueHost should know about this, it's a little strange that the person you spoke too did not know what to do or transfer you to someone who knew more.

Link to comment
Share on other sites

This is a good thread for info, Dolke, a lot of your posts relate to me as i believe I am the the same stage as you, I am just preparing CSV's now for upload and a lot of the the problems you have had so have I.

 

i think a lot of the info is out there, just hard to access/find, but the mods on here are all very knowledgeable and helpful. Also need to remember this is open source software and people offer their help and advice for free.

 

Update me on the VPS performance difference as I will be setting that us soon as well :)

Link to comment
Share on other sites

As for how much server power you need, it all depends on several factors like how many products you have, how many visitors your site is getting a day, and how optimized your site is. One thing you can use to give you a little insight is loadimpact.com they use AWS to create virtual users and load down your site to see how it operates under load.

  • Like 1
Link to comment
Share on other sites

Dh42, another great tool you gave us.

 

Sound good. I will check that as soon as I completely transfer to VPS.

 

I have around 3500 products and at this moment I'm not getting more then 50-70 visitors per day.

 

When it comes to optimization I haven't done anything yet, only what Prestashop offered me in the first place.

I know that there are lot room for improvements, but I have clear goal. Make the website running and make it possible for visitors to order things and then I will make up each and every corner of the website step by step.

 

I think it is the right way to do it...

Edited by Dolke (see edit history)
Link to comment
Share on other sites

I am reporting back, but the report is not good :-/

 

So I did what DH42's blog post recommended. I checked the info.php page- the changes went through. Then when I tried uploading a 270KB CSV file, it timed out (went to the 404 error page) at the same point (85 products in). So I tried refreshing the page (the prestashop 404 error page that appeared when the back office csv complete page usually appears.) I refreshed the page a few times, in case it was just picking up at the point the CSV upload had timed out.

 

That didn't work. It had only completed the original small chunk. I don't know if it was just rewriting the first section over and over each time I refreshed the 404 error page or what.

 

So, like Benjamin said, I talked to Bluehost again. DH42's blog post/ recommendation said some hosts wouldn't accept the ini file. Bluehost assured me that the php.ini was acceptable and did belong in the root folder where I had put it. Bluehost made sure I had the right version of php- 5.3. I sent them the link to DH42's code to double check that Bluehost's system should work with it. She tried some other stuff, but in the end just told me to delete the cache and cookies, and try a different browser.

 

I did this. First in chrome, then in explorer. Explorer appeared to have more problems than chrome.

 

I know I can divide the inventory into smaller chunks if needed, but that probably won't stop the inexplicably different results each time I upload the same files. The way the categories upload has been changing around, all while I'm using the same spreadsheet as a basis. A subcategory has been appearing at the top menu bar sporadically while root level categories won't appear at the menu level. I thought that is what a root category meant- it appears at the top level menu.

 

This whole new slew of issues is very frustrating. I reupload a spreadsheet, and it may or may not do the same thing. One time the breadcrumbs for a subcategory were listed with the wrong category, though it was actually in the right parent category. Then products refused to load into their assigned category and just went to home. (I double checked the spreadsheets and can't figure out why.)

 

I'm sure I'm making some mistakes here, but I don't think that is the only thing at play.

 

If anyone has any ideas or has experienced similar issues either with the timing out during csv loading or general csv loader bugs please let me know. In the mean time, I will sift through the forum for bugs in 1.5.4.1, but if anyone has had the same thing happen, please point me in whatever direction you think would help.

 

Helen

Link to comment
Share on other sites

How long does it take before you see the 404 page? It should be about 10 minutes. See the way a php.ini works is that not all hosts will let you override all values. If the webpage is not loading for something like 10 minutes, it means they are not letting you override the execution limit. Can you send me a pm with the location of your info.php file.

  • Like 1
Link to comment
Share on other sites

Dear Helen,

 

I believe, as Dh42 said, you cannot override these limitations if you are hosting in a shared hosting environment.

Most of the web hosting companies wont allow this. I host at HostGator and even now when I'm on the VPS, they are telling me that it is not recommended to put such high values.

 

I would suggest that you contact your hosting provided and give them the link to mr Dh42 blog and post so they can check can you or cant you do these corrections in php settings.

 

If you cant, than you need to migrate to VPS or to break csv file in smaller files like I did. I was using 100 products per file, not more.

  • Like 1
Link to comment
Share on other sites

Just to add to this, i am testing PS on the most basic shared hosting package ( Its like £2 per month! ) and will upgrade to VPS once testing is complete. The Host I am using is TSO Host, a UK company and they allowed my to change the settings as per DH42's specification with no issues.

Link to comment
Share on other sites

Thanks for letting me know. I was getting the idea from my host that unless I upgraded, increasing some of those limits (from DH42's post) wouldn't be possible.

 

For now I'm just going to decrease the size of a bunch of the photos and upload in smaller sections. I can work around it for now and deal with switching hosting in a while. It looks like the 64MB memory limit with shared hosting at Bluehost could be a problem running the newest version of prestashop at some point. Not sure about that one, but it seems like prestashop 1.5. needs at least that much, and that is the max with shared hosting. DH42 was recommending 2048MB, so I could imagine with large shop large/ complex shop that might be a problem. That is mostly just a guess...

 

By the way, the host's response to my questions about getting the php.ini file to increase the limits was:

No the .htaccess handles all that for each sub domain you should see in the file

# Use PHP5.3 Single php.ini as default

AddHandler application/x-httpd-php53s .php

however is you wanted to use a different version of php on a sud domain you can the php.ini

& the .htaccess must be in the directory.

 

 

Ony a part of that means something to me. The recursive application of the php.ini with php5.3 is what I was asking about. Thought I would throw that out there in case it meant more to one of you.

 

As always, thanks for the help, all!

Link to comment
Share on other sites

It wasn't that prestashop needs 2048mb memory, I just always throw a lot of memory at it. The main problem with your import was the image size, you are using really large images which takes a lot of time.

  • Like 1
Link to comment
Share on other sites

Hello folks,

I strungled with The Same prob, max 100 Products at 1 cvs import run, I received allways a white blank screen, I host by one.com, the max server connection time are 30 sec.

I run PS V1.5.4.1

 

Succ6

Generaal

Link to comment
Share on other sites

I'm on the VPS now but it seems that problem remains.

I really can beleive that even VPS server cant handle this.

 

What I got from the HostGator is that I will need at least level 6 or even dedicated server in order to pull such a massive resources that Presta needs when we use these options. Same problem of breaking happens when I run indexing of my images, but not partial indexing but building the whole index from the beginning.

 

I was also told by the HG representative that on the shared environment I have more resources then on my VPS but I cant changed php ini setting while on the VPS I have that option but I dont have enough resources.

 

Bottom line is that I'm pretty much in the same position on this specific VPS account then I was on shared except I had email problems, I payed extra and wasted around 2+ days...

Link to comment
Share on other sites

I'm on the VPS now but it seems that problem remains.

I really can beleive that even VPS server cant handle this.

 

What I got from the HostGator is that I will need at least level 6 or even dedicated server in order to pull such a massive resources that Presta needs when we use these options. Same problem of breaking happens when I run indexing of my images, but not partial indexing but building the whole index from the beginning.

 

I was also told by the HG representative that on the shared environment I have more resources then on my VPS but I cant changed php ini setting while on the VPS I have that option but I dont have enough resources.

 

Bottom line is that I'm pretty much in the same position on this specific VPS account then I was on shared except I had email problems, I payed extra and wasted around 2+ days...

 

Damn,

 

That is no good.. i have just uploaded my first CSV, of only 130 products. I had change the config on the server as per DH42 instruction, uploaded CSV, seemed to be going ok, then

 

Internal Server Error

 

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, [email protected] and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

 

Only about 60 of the products are in BO...what to do now, I don't know why it failed or at what point..

Edited by RCP90 (see edit history)
Link to comment
Share on other sites

RCP90,

 

Your upload timed out. Can you confirm in your php.ini setting that the execution time was increase? Also, how long did it take for you to get that error?

 

If you can increase your execution time, then even if the page still times out, your upload can still process in the background.

 

 

Dolke,

What vps package do you have?

Link to comment
Share on other sites

I just contacted the host and they will increase the execution time some more, and will confirm how long. I am only using shared hosting atm, but they allow custom changes to php.ini, so they i changed the settings as you suggested.

 

I got the 500 error after a couple of minutes.

Link to comment
Share on other sites

Ah OK,

 

They just replied and said that it is currently set to 5 mins. I can get 60 products in that time, but then will 500. I will see if they will increase this to 30mins.

 

Getting issues with features not importing now :/

Link to comment
Share on other sites

This morning i was on HostGator VPS level 3 and got upgrade to VPS level 5 and I'm still struggling.

 

Something is seriously wrong or with Prestashop or with the Hostgator. Support person told me that my website is overloading the server constantly...

 

Here are my resuilts from the GTmetrics: http://gtmetrix.com/....co.rs/bJKcYXcS

 

They are also telling me that my website needs optimization. They are right but how come I get better performance on the shared hosing then on the 100$/month VPS?

 

btw Dh42, your website http://neat42.com/prestashop/index.php is super fast compared to mine, http://www.dekom.co.rs

Edited by Dolke (see edit history)
Link to comment
Share on other sites

Also I'm on the line with them and I want to enable mcrypt and memcache php extensions on the server.

Maybe that will decrease my average load time per page which is ridiculously high, 28 seconds :huh:<

 

RCP90, what issue are you getting when importing? If they are CSV related maybe I can help as I had similar issues but then figure some of them out and finally manage to import smaller chunks of main CSV file without any errors, or just with some less important warnings.

Edited by Dolke (see edit history)
Link to comment
Share on other sites

Dh42, my website worked better on my shared hosting plan then on this VPS lvl 5 that I currently have at HG.

I doubt that resources are problem here. It is something else and I cant figure out that it is...

 

I have no reason to think that it would work differently on that VPS plan you posted. I upgrade and they keep saying that I need more resources as php script is overloading the server... I believe that they will told me the same story if I'm on the dedicated server too

Link to comment
Share on other sites

No Sir, I still have the same problem. I tried on both VPS and shared hosting...

 

At this point I'm trying to find out, to pinpoint, what is actually causing this, what kind of resource is involved.

Is it CPU, is it memory limit, someother...?

 

P.S. What I heard and learned from the Hostgator.com so far is that their VPS is designed to use as a testing server environment and not a real solution for hosting websites. Also their shared servers are much more powerful machines then VPS. Also Prestashop doesn't work ok with their VPS (I went all up to level 5 VPS).

 

This particular import problem doesn't work on the shared hosting as well as on their VPS, but on the VPS some other issues appear that are not present on the shared environment probably due to more powerful servers.

 

Error log on the server is showing errors while I try to import products. This was confirmed to me by Hostgator support agent.

Edited by Dolke (see edit history)
Link to comment
Share on other sites

Hi again,

 

Most of the users are knowing the following settings;

memory_limit

upload_max_filesize

max_execution_time

max_input_time

etc...

 

If you created your .csv file properly and if the above settings ok, but still not solved your 404 errors and/or internal server errors,

 

I suggest, disable mod_security module and try to import process again.

Probably it will be imported correctly.

 

But as I know, mod security is the most important firewall for your web site. Its preventing from the attacks.

So, you must make some changes in mod security rules instead of disabling it.

Link to comment
Share on other sites

I don't think the 404 or internal server errors can be mitigated. They are not really a problem per se, they are just a product of something taking a long time. Most of the time they are returned because of the loading time limit that a browser will try to load the web site.

 

the only real way around them would be to change the whole import process to use ajax instead of posting it.

Link to comment
Share on other sites

Hi guys,

 

I have the same problem and have had the same issues ever since PS 1.3. It seems with every new PS version the same problem with importing csv crops up! this is ridiculous. I got it working before with PS 1.4 by changing the adminimportcontroller code from github but now the same thing is happening again with 1.5.4.1.

 

I have changed the coding of adminimportcontroller with the github ones, i have changed the php.ini and uploaded it to my server as mentioned on the dh42 blog, I have spoken to Hostgator too but I keep getting the same issue with importing csv files. Hostgator recently migrated and upgraded their servers but they assured me this has nothing to do with it.

 

I even tried to break it up into smaller sections but i keep getting an error that the server closed the connection without sending information

No data received

Unable to load the webpage because the server sent no data.

Here are some suggestions:

Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection without sending any data.

 

or I get the bloody 404 page or the "shop is in maintenance" page!.

 

My one csv file has 1400 rows, 3 columns, contains only text (no images etc) and is only 13Kb! so why the hell is this happening?

 

Needless to say I am so bloody frustrated with this as I spend more time trying to "fix" my website as actually trying to run my shop and sell my products (which is what Prestashop was supposedly designed to do!) I am not a coder Prestashop, I am a retailer trying to sell my products, so please do what you guys are supposed to do and get the coding done so I can focus on what I want to do! For once I just want my shop to WORK without me having to spend days trying to figure out why it's broken.

Link to comment
Share on other sites

Hi to both,

 

@tuk66

 

It is not hosting plan. I tried going on the VPS and had even more problems than on the shared hosting server. Also, a friend of mine has MS Azure cloud, and he made me virtual machine with 8 gbs of ram, with quad core processor and import was still behaving the same.

 

@I_raath

 

I completely feel your frustration as I have the same problem. I also started this topic. But mr. tuk66 does have a point.

Try to be precise and narrow down your post as people here can get lost in the longs posts and miss the point you were trying to make.

 

 

I would like to ask Mr Benjamin to step forward and investigate is this actually bug in the PS and it would be great to pin this topic to the header as I noticed many people here experience all kind of import problems.

 

Thanks

Link to comment
Share on other sites

  • 4 weeks later...

Hey Dolke,

 

I've been experimenting with csv files for some time now and I created a csv file that i can import onto a local server that is in my office without any issues. I've uploaded about 15 thousand combinations and about 4000 products without having to split up my csv file. On godaddy though, it times out so I did it all locally, and took a database dump and imported the database to godaddy (which works just fine).

 

On another site I only had about 170 products and these imported successfully on my local server. But when I tried to do a csv import on godaddy it initially imported successfully until I realized that i had the wrong url address for the images. I changed the url to the correct address and reimported, this time it timed out on godaddy. I went to go look at the image name and I noticed that one of my employees had put a space in the file name of the image. I removed the space and edited the file name and re-imported. This worked but I did run into a few other smaller issues, which i eventually chalked up to being a time out issue on godaddy. I ended up having to just split my file into two and importing them separately.

 

I'd check your image URLs just to ensure that there are no spaces in them and see if that helps anything. It helped me....

Link to comment
Share on other sites

@shiggidydog

 

Thanks for sharing the information. I stopped with this investigation as I already made import (by splitting to smaller CSV files) and I'm managing my store using Store Manager desktop app now so I don't really use CSV import functionality anymore.

 

I doubt image paths are cause for the problem. My argument for that is simple: I never received any errors regarding paths or images while importing. If Prestashop doesn't report the problem and it is happening than it is bug in the prestashop right.

 

Please, share your thoughts about this issue. Maybe I can politely ask mods to pin this topic as it is very widely problem.

 

Thanks

Link to comment
Share on other sites

  • 1 month later...

I also had the same problem, the csv import wasnt imported all of my products. Now it wont import at all. I get to the 'view your data' page and there is no data to see even though im using the same spreadsheet as before when it was working but not importing everything. Can any help as to why the csv is not working at all now please?

Link to comment
Share on other sites

@shiggidydog

 

Thanks for sharing the information. I stopped with this investigation as I already made import (by splitting to smaller CSV files) and I'm managing my store using Store Manager desktop app now so I don't really use CSV import functionality anymore.

 

I doubt image paths are cause for the problem. My argument for that is simple: I never received any errors regarding paths or images while importing. If Prestashop doesn't report the problem and it is happening than it is bug in the prestashop right.

 

Please, share your thoughts about this issue. Maybe I can politely ask mods to pin this topic as it is very widely problem.

 

Thanks

I agree. The problem is that many huge tasks just fail silently without any relevant information for user as "divide your import to smaller parts because you reached server timeout limit".

Link to comment
Share on other sites

  • 1 month later...

Hi,

 

we had the same problem with too many articles but.... I've got an expert in the family.

 

My son just programmed a script for me that splits the product-CSV and organises the import of the articles selfstandingly.

 

What you would need:

- ssh or telnet access to your shop server (Linux server)  to start the script

- CSV products file as per prestashop definitions.

- could be started by cronjob

 

It's now running in the first version, will still need a bit of fine-tuning: I'm now importing all 800 articles with pictures which failed at 110 before.

 

He's student in the first year but he probably would love to share the script for a little Amazon gift voucher by mail :)

Link to comment
Share on other sites

  • 10 months later...

I agree, I think that maybe you should call again after reading Dh42 article. It is concerning the php.ini file and variables like max_execution time. BlueHost should know about this, it's a little strange that the person you spoke too did not know what to do or transfer you to someone who knew more.

Yeah right and who will you blame if I told you that it takes 6 minuts to import 800 rows on a dedicated server in this format:

 

536403-001;"Power Supply 400W";;HP;536403-001;5704327785132;"AC Adapter";;0,00;1179;0,40

 

CVS import function is really poor in need of being rewritten.

Link to comment
Share on other sites

×
×
  • Create New...