Prestashop Import Process

[EcoPets]

I do not know if this is the most effective place to put this but I can't seem to find anywhere else where my opinion will be heard and taken seriously.  There is no "Complaint Dept", maybe by design? So here goes. 

The entire product import process stinks!  It was poorly thought out, poorly designed and poorly executed!

First, a little on my background.  I have been working with computers and software since the mid 70's.  I created business software in Cobol, Assembler, RPGI & RPGII on the IBM 360.  There is more memory in your smart watch than that 360 had, so we had to write code that was tight and efficient.  Today, that principle is totally gone and platforms such as Prestashop are guided more by the financial push to get to market first and work out the bugs later.  Then users like us become the test bench and we find the bugs and figure out ways to fix them.  And for the most part, that generally works out OK, until the next update!

But this Import process seems to have been totally misconceived from the beginning.  From what I can tell, it never has worked effectively or efficiently.  I believe that is because the teams(s) that produced it know next to nothing about product record keeping or inventory processes.  Along with the 30 years working with software, I spent more than 40 in the wholesale supply industry dealing with and maintaining inventories worth millions of dollars, retiring about 10 years ago.

I have spent the last several days trying to import an Excel spreadsheet of about 8000 products into Prestashop 1.7, 8.0 and 8.1  The import has failed EVERY TIME for a variety of reasons.  After dozens of attempts, I KNOW what fields to map and when I do it on a file with only a few items it generally goes fine.  But when I try the full file, 3 significant things that happen that are completely unacceptable!

1.  The import fails and the message is "an unexpected error occurred" - there is no mention of what cased it or where.  It just failed.  Sometimes the logs give some idea of what happened, sometimes not.  A number of searches finds that I am not alone in these issues and sometimes there is an answer and most times that is a hack into some file and/or part of the database that is only good until the next update.  THAT is not a solution!!  That is a band-aid!

2. The import fails part way through with some sort of error about an entry or field from the import that is somehow invalid.  And in an import of 8000+ records, I would expect that there would be a few items that would throw an error, BUT, and here is what REALLY is unacceptable, all of the records that were correct before the error and passed without issues are now not even loaded in the database!!  In everything I have worked on over the past 40+ years, those records should have been intact and if the process failed on 1 that stopped the import, then we see what that record is, work out how to fix it and then restart the import FROM THAT POINT and continue until we get all records loaded!  It really is that simple!  But the way Prestashop has chosen to handle it is to throw an error and not provide a way to correct it or go around it, throw out the entire process and make the user start all over again!  And even if you fix that one record that stop the import, in 8000 records, it is a pretty safe bet there will be more!  So you are doing the same thing over and over again and expecting different results!  That just wastes users time, and I have wasted A LOT of time over the past few days and STILL do not have ANY the products loaded in the database!

3. There is this REALLY useless little section at the beginning of the import where you are actually supposed to be able to save the mapping configuration you setup for that import under the really smart premise that then you can retrieve that configuration and use it again on the next import of that sheet, which would actually save a LOT of time!  BUT this is where that rush to market creates a big problem!  While you can give that configuration a name and actually save it in the data base, the table is PS_IMPORT_MATCH, and you can DELETE a configuration from that table if needed, the one most important part of this little tool, the one that says "LOAD", simply does NOTHING!  So some coder just did not finish the process!  My instructors would have failed me on that one point!  My bosses would have fired me on that one point!  My clients would have fired me!  But that problem has persisted, as far as I can tell, for about the last 3-4 versions!

I have actually seen posts in these forums on these very issues going back at least 7 years!  And the developers STILL have not correct the problems?  Now maybe I am just shouting into the wind here and the usual result will be that nothing will happen.  I know that some enterprising and talented user will put together some module to fix the problem, AND charge a pretty penny for it!  But the fact remains that Prestashop without a good, complete and efficient inventory system is nothing more than a door stop!!  MAYBE, just maybe one of the Prestashop development team might read this and decide to do something about it.  I'm not holding my breath tho!

[EcoPets]

Update:

Using 1.7.8.10 on my local machine, I was able to get the LOAD configuration to work and save the tiresome task of mapping the fields each time.   I cannot seem to get it to work in any version 8.  As it was I had about 8000 products to import and it took about 10 tries to get all of the "invalid" or "field length" etc cleaned up in the vendors product list.  BUT, after I got all of that done, I split the file in 2 parts to maybe over come any timeout issues, and the import went fine, at first!  The first pass import does is to validate the entries and that ran thru at 100%.  Then import proceeds to import the records.  As you can see in the attached, it got to 80% and then stopped and reports "Errors occurred: error"!  So helpful!!  It took about an hour to reach this point so that is about 3600 seconds.  Since this is on my localhost, I can adjust those PHP settings and see what happens .  Now I KNOW that the spreadsheet is 100% correct!  It took around 5 hours to clean it up after repeated stops because of some issue.  There are NO error logs on the server that show ANY errors when this happened.  I got about 3500 products imported before the process stopped.  Now, I have no idea what stopped it and the developers did not see fit to put in ANY sort of helpful error that might give me an idea of where to start!

[musicmaster]

Look in your server's error log or under the directory /var/logs for the error messages.

On Prestashop's Github you can file issues. Most of Prestashop's developers don't visit this forum. Be kind to them: they are volunteers.

https://github.com/PrestaShop/PrestaShop/issues

[EcoPets]

my local wampserver logs do not show any errors.  I ran it again and this time it stopped at 69% with no helpful errors.  It might be a timeout error but I have PHP setting at 5000 minutes and that should be plenty for this to run.

[EcoPets]

Update:  This is definitely at timeout issue.  I reran the import with a product file size that is 25% of the original.  The import completed in about 45minutes.  The full file appeared that it would take about 3 hrs.  When I cut that size in half (previous import), that appeared it would take about 1.5 hrs.  So the time out seems to be about an hour.  There is not documentation of this any where.  So I am wondering if anyone else has faced this issue with around 8000 products?

[Danmant_DK]

On 8/23/2023 at 3:46 PM, EcoPets said:

Update:  This is definitely at timeout issue.  I reran the import with a product file size that is 25% of the original.  The import completed in about 45minutes.  The full file appeared that it would take about 3 hrs.  When I cut that size in half (previous import), that appeared it would take about 1.5 hrs.  So the time out seems to be about an hour.  There is not documentation of this any where.  So I am wondering if anyone else has faced this issue with around 8000 products?

In earlier versions i'v got the same problem. In vs 1.7.8.10 i can import/update 26000+ products without issues. I always keep the ID in the beginning of the csv file

[EcoPets]

once I get all of the data irregularities cleaned up at the CSV properly formatted, it will import but the the files, now about 6000 products still takes about 2 hours.

[Danmant_DK]

1 hour ago, EcoPets said:

once I get all of the data irregularities cleaned up at the CSV properly formatted, it will import but the the files, now about 6000 products still takes about 2 hours.

You can possibly try to remove some columns that you suspect are extra time-consuming and then later import them separately.
In addition, my speed was reduced by PC, too little server memory and not optimal PHP version (I use version 7.4.33 today together with PS 1.7.8.10)

[EcoPets]

That is part of my cleanup process.  I make the CSV as concise as possible to still get all of the necessary details.  I had tried working with PS 8 but find that many so called " improvements" actually created more problems.  So I am staying with 1.7.8.10 as long as I can until PS 8 is working as well as 1.7.8.10.  I know that sooner or later PHP7.4 will cause problems, but for now it is the best solution I have.