[SOLVED - for now] Duplicate images during CSV Import

[uddhava]

PS : 1.3.1

I am using csv files to import products.
When i re-import to update the products all the product images get duplicated.
In the image directory multiple images appear for the same product like this :

701-400.jpg
701-440-home.jpg
...
And the duplicates

701-448.jpg
701-448-home.jpg

And when i try another import even more images are created..
What's going on here??
I suppose the ID is not checked properly during import.

I cannot understand that nobody else is experiencing this? Or am i really the only one?

[rocky]

I've noticed that too. You can't delete images from the CSV import, only add more images. The only way to delete images is to tick the "Delete existing products?" option, but then you must always have all your products in the CSV, you can't import just some products. You could try posting this issue in the bug tracker. If you're lucky, the PrestaShop team will fix it, otherwise they will probably say "post it as a feature request".

[uddhava]

My CSV files are divided into categories. This makes it easy to update and then i know where to find stuff. Also i use fixed ID's and a maxium of 200 ID's per file to decrease the import time.
When i select "Delete all products" during import it will delete ALL products and not just those i re-import.
I think the image check is faulty and should be fixed soon.

I have submitted a Bug report for this. Hopefully it will be addressed because i found a similar bug report that was reported for PS 1.1.....
In this bug report it was mentioned that the image check function does not always return a correct answer and so images get duplicated. (see thread)

The creation of duplicate images would not be such a problem, but because in the front end each image is displayed it is a big problem. For some articles i have 6 images. And deleting them is only possible through the back-end, and one-by-one...

There was a funny suggestion from the PS team. When doing a re-import do NOT include the image url again....... sigh

[uddhava]

He Rocky, it looks like you can look into the future. I got a reply to my bug report and they say it is "expected behaviour". And if i want to change that i have to file a feature request.

But for the general public (and also myself) i was wondering;

During an import of products every image file is uploaded, "decoded", copied and renamed into the /img/p folder.
Even when you re-import, the same file gets processed.
Unfortunately (or rather expected) every image is displayed on the front end later.
And every image gets an ID and other number in the file name. I guess this is a smart way of handling multiple image files for 1 products.
And since PS does not "know" the difference between images, it will always duplicate images...

The easiest way around this would be that when the import starts, the ID in the CSV is checked and then the corresponding images are deleted.
That should be possible, and i will try to get this feature worked out.

But as a headstart i would like to see if i can program this myself, to test it out. But i cannot seem to figure out how the image processing is done and where. So bare with me;

* AdminImport calls -> Import Class, AdminTab and images.inc.php
* AdminImport checks images with function : getImages() and productHasImages()

And then i'm lost....

[uddhava]

I found the order of importing images during AdminImport. (/admin/tabs/AdminImport.php)

When importing a CSV, the image(s) gets checked & uploaded by this function (from line 719)

if (isset($product->image) AND is_array($product->image) and sizeof($product->image))
               {
                   $productHasImages = (bool)Image::getImages(intval($cookie->id_lang), intval($product->id));
                   foreach ($product->image AS $key => $url)
                       if (!empty($url))
etc etc etc

This code will trigger the copyImg code :

{
                               if (!self::copyImg($product->id, $image->id, $url))
                                   $this->_warnings[] = Tools::displayError('Error copying image: ').$url; 
                           }

The copyImg function is also found in the same file on line 374.
This will upload the image and resize it. But it does not check if there are already images. So i added some code that calls the function deleteImage (that is in images.inc.php on line 251.
The new code looks then like this :

if (@copy($url, $tmpfile))
       {
           deleteImage($id_entity,$id_image);
           imageResize($tmpfile, $path.'.jpg');
           $imagesTypes = ImageType::getImagesTypes($entity);
           foreach ($imagesTypes AS $k => $imageType)
               imageResize($tmpfile, $path.'-'.stripslashes($imageType['name']).'.jpg', $imageType['width'], $imageType['height']);
       }
       else

I tried it but it will not delete the existing image...
Have to dig further. Any clues?

[rocky]

Try changing line 515 of admin/tabs/AdminImport.php from:

$product = new Product(intval($info['id']));

to:

$product = new Product(intval($info['id']));
$product->deleteImages();

This code should load the product using the ID specified in the CSV, then delete all the images on that product.

[uddhava]

This almost works.
The previous images get deleted and new image(s) are created.
But the image is not shown in the front end. The image is present on the product page (back end) but it is not set as the "cover" image.

I tried again. First delete the whole product and try again with your code. Now it works...
Doing a re-import also works..

Now i will try a big batch.

[uddhava]

OK i tried to import a big batch.
Now all the images have disappeared from the front end. The previous product images are deleted, new images are created, but the "cover-setting" for the new images does not take place.

When i delete the products first and do an import the images will be displayed.
Another re-import will mystically let the images dissappear again.
So something happens in the $product->deleteImages(); procedure that resets the cover image setting.

Now that i am thinking about it, it might just be easier to delete the whole product before importing the new one.
Then the images get deleted also.
I am guessing the code will be something like this : $product->delete();

Correct ?

[uddhava]

Hahaha... I tried my delete code and now the product is deleted... go figure :-)

[uddhava]

Anyway, i cant figure it out at the moment.
I either want to delete the images or the whole product based on the product ID.

[uddhava]

I am so HAPPY...
I added some more code and now it works. I found out that when a product has images (which is always the case during a re-import) the image cover bit does not get set. To overcome this and to delete all the previous product images i have changed the AdminImport file.

The new code below will delete all images for the current product (ID) and then import the images from the CSV file. If you have specified multiple images, it will import them all, but only the first one gets set as the "image cover".

Here is the new code : (/admin/tabs/AdminImport.php from line 720 - PS 1.3.1.1)

if (isset($product->image) AND is_array($product->image) and sizeof($product->image))
               {
                   $productHasImages = (bool)Image::getImages(intval($cookie->id_lang), intval($product->id));
                   $coverset = false;
                   foreach ($product->image AS $key => $url)
                       if (!empty($url))
                       {
                           $image = new Image();
                           $image->id_product = intval($product->id);
                           $image->position = Image::getHighestPosition($product->id) + 1;
                           if (($productHasImages) AND ($coverset == false))
                               {
                                 $product->deleteImages();
                                 $image->cover = true;
                                 $coverset = true;
                               }
                           else
                           $image->cover = (!$key AND !$productHasImages) ? true : false;
                           $image->legend = self::createMultiLangField($product->name[$defaultLanguageId]);
                           if (($fieldError = $image->validateFields(UNFRIENDLY_ERROR, true)) === true AND ($langFieldError = $image->validateFieldsLang(UNFRIENDLY_ERROR, true)) === true AND $image->add())
                           {
                               if (!self::copyImg($product->id, $image->id, $url))
                                   $this->_warnings[] = Tools::displayError('Error copying image: ').$url; 
                           }
                           else
                           {
                               $this->_warnings[] = $image->legend[$defaultLanguageId].(isset($image->id_product) ? ' ('.$image->id_product.')' : '').' '.Tools::displayError('cannot be saved');
                               $this->_errors[] = ($fieldError !== true ? $fieldError : '').($langFieldError !== true ? $langFieldError : '').mysql_error();
                           }
                       }
               }

If you dont want to edit your file then you can replace the AdminImport.php with the one below.

AdminImport.php

[rocky]

Thanks for posting your solution. If this issue is resolved now, please edit your first post and add [sOLVED] to the front of the title.

[uddhava]

He rocky, many thanks for the hint that lead to succes.
But my programming is a bit clumsy i think. So i hope someone / PS team will take it up to write a better solution then the one i did.

ys

[SandyW]

uddhava, you are great! Thanks a lot for sharing with us what you've got!
Great help

[Adolfo_OM]

I install the changes, and it works fine when import the products, but when I try to add an extra image edditing the product by hand (backoffice >> Catalog >> ...) I can't add any extra image.

Any idea?

[tester123]

uddhava, fantastic solution. Thanks.

[miki22]

How to remove images duplicated while import? I mean overall removal of all files, no just product by product.

[tester123]

Usually all product images are kept in /img/p/
By name of a file You can tell to which product it belongs /all are numbered and specified if its thickbox etc/. So delete those and upload products with urled images again. This way You update whole image set for each product.

Now, this is my theory, so test it before You go wholesale.

[miki22]

Usually all product images are kept in /img/p/
By name of a file You can tell to which product it belongs /all are numbered and specified if its thickbox etc/. So delete those and upload products with urled images again. This way You update whole image set for each product.

Now, this is my theory, so test it before You go wholesale.

I did test.
When removed files from /img/p for id of product and updeded by import i got 7 empty images of product [? on img] and 8th img with correct foto.
I will try Regenerate thumbnails in preferences/image/ but i got 2000+ products and it will take some time.
And thath may not work properly.
Any help? Is there some SQL query to update product?
I know that prestaclient helps for fast update of products and i try to find it in the web but
it will be greate if somone will do some module or update to PS to multiple imgs remove.

[tester123]

OK.

Other solution.

Upload the products with image urls as usuall. But before You do it upload the script attached. I sourced in on this forum, its been created by one of the guys in this thread. It adds tick option to upload window: Delete old images before upload. You should be able to delete images /if any still there/ and do changes in code so wont get "?" image.

I havent used that script yet, I am still preparing 2k database /brrr/, but its been published by presta as official solution to a bug.


It would be usefull if You create a test copy of PS for all Your trials, just in case all goes to hell due to human mistake.

AdminImport.php

[miki22]

OK.

Other solution.

Upload the products with image urls as usuall. But before You do it upload the script attached. I sourced in on this forum, its been created by one of the guys in this thread. It adds tick option to upload window: Delete old images before upload. You should be able to delete images /if any still there/ and do changes in code so wont get "?" image.

I havent used that script yet, I am still preparing 2k database /brrr/, but its been published by presta as official solution to a bug.


It would be usefull if You create a test copy of PS for all Your trials, just in case all goes to hell due to human mistake.

The tick is:
Delete all products before import ?

I want to delete images not products.
But even if so, I don't want to delete all products but only imported. IDs fro 200 to 1000

[pcitshop]

this scrip adds duplicate product

[ishant]

Hi All,

Need your help.

I am using this extension http://codecanyon.net/item/prestashop-import-product-with-csvexcel-module/12234453 to upload the csv products. Issue is: I uploaded 10 products but its uploaded 80 products in the database.

Please help me to come out of this issue.

Thanks and Regards,

Ishant Arora

[susieb]

Hi - I'm having this very same problem but in 1.6. I am not technical but could change the file over if I could find the adminimport.php file - think it must be different in this version. Can anyone help with this. I need to remove all of the duplicated images in bulk and can't seem to find a way to do it.

 

Thanks in advance 

[yguerin]

Dear,

 

for prestashop 1.6 GOTO = https://www.prestashop.com/forums/topic/530956-php-send-delete-existing-images-during-import/

 

Regards,

 

Yves

[sandeepintuitm]

hi guys,

 

i am uploading products from csv. I have image urls like "../userdocs/skus/p_006105000_1.jpg" but it is not accept this type. How can remove this  validation from prestashop code. 

[yguerin]

Dear sandeepintuitm,

 

Try to add the full path to your image url: file://path/to/your_image.jpg or something like this, I use http://my_address_ip:81/path/to/my_image.jpg in my csv file and on the host my_address_ip I created a virtual host pointing to the filesystem where my images reside.

 

I able to send you my setup files if needed

 

The jpg is recognized by prestashop, I use jpg too

 

Regards,

 

Yves