Jump to content

[MODULE] Fast CSV or XML import of thousands products in a few seconds


presta-dyr
 Share

Recommended Posts

I offer a module that can import XML (and CSV) files. I have been working with files that import thousands of products.

One of the big problems when importing is that resizing the images takes way too long time. Typically the server can resize only 1 image per second. I have found a way to handle this.

Features
* Fast import of products from CSV og XML files.
* Support for multiple languages.
* Support for multiple images.
* Support for multiple suppliers
* Support for attributes.
* Support for features.
* ISO-8859-1 or UTF-8.
* Possible to automatically calculate the sale price based on wholesale price.
* Possible to specify delimiter for categories and sub-categories.
* Option that allows alphabetically grouping of sub-categories (useful if there are a lot of them).
* Installs as a tab under "Tools" in the back office.
* Possible to resume the import if the maximum run time is exceeded.
* Automatically creates categories and sub-categories.
* Setup for mapping of CSV or XML fields are saved in the database.
* Images are fetched and resized on demand.
* A cron job can handle the resizing of the images at nights.
* Comes with manual in PDF format.
* Possible to update existing products.

Notes
* Products, categories etc. should not be deleted but you may automatically deactivate products that are no longer in the CSV/XML file you get from your supplier.
* It is possible to delete all images from a specific supplier from the back office.
* It is possible to delete all products from a specific supplier from the back office.
* It is possible to delete all empty categories.
* It is possible to delete all products, categories etc. if you really want to start all over.

If you have a lot of categories, PrestaShop (1.2 and 1.3) tends to load slowly. I have posted a fix for this.

The price is $100. You can buy the module from my shop.

I will no longer offer a demo (I spent way too much time managing my demo site). Instead I have added some demonstration videos (look in the bottom of the description).


Edit 2010-10-20: Updated with new features like multi-supplier.
Edit 2011-05-21: Works with PrestaShop 1.4.
Edit 2012-08-05: Works with PrestaShop 1.5.
Edit 2012-11-02: Added link to shop.

Edit 2015-05-06: Works with PrestaShop 1.6.

/Kjeld

Edited by presta-dyr (see edit history)
  • Like 9

Share this post


Link to post
Share on other sites



Notes
* All products, manufacturers and categories are deleted when importing.

The price is $100.
/Kjeld


Nice module, but what about if someone needs only to import products into existing categories?

Share this post


Link to post
Share on other sites

Sure, I have a bunch of (authentic) CSV and XML files I have used for test. I guess you are looking for something specific? I mean a simple, short CSV file works fine with the existing import function.

/Kjeld

Share this post


Link to post
Share on other sites

Actually, there is an option for exactly that (i.e. preserve categories).

Cheers,
Kjeld


Is there an option so that you can only add products?
I have my categories and I have 500 products which I need to keep them.

Share this post


Link to post
Share on other sites

Currently it is not possible to add products to an existing base. It is much faster just to add products from scratch than to update products. But I have an idea how it can be made...

/Kjeld

Share this post


Link to post
Share on other sites

Presta-dyr

look at my post : http://www.prestashop.com/forums/viewthread/58071/job_offers_and_paid_services/i_need_a_xml_importer

two files (one xml example, other is a field setting document)

can your importer do this?
ps : product can NOT be deleted when starting import.
only when the status code <>"NORMAL" then you can delete a product.

this because I have two suppliers.
one small (over 450 articles) I've added manualy those products.
the other over 15000 articles I must update/delete/add every week/night.

look at my post and I'll hear it back from you.

Share this post


Link to post
Share on other sites

Hi deech123.

Currently the module does not support appending or updating products. But if you're willing to pay for it I can make a special version that suits your needs.

I have sent you an e-mail.


/Kjeld

Share this post


Link to post
Share on other sites

I'll be interested too in this module supporting appending and updates; so far i haven't had the time to fully test this module due to the countless problems I have after upgrading presta to last version; I hope that I'll finish this in a few days and get back to testing because god knows I'm in great need of a such working module.

Share this post


Link to post
Share on other sites

I have submitted a number of modules (for free) and have spent a lot of times translating PrestaShop to Danish. I do my part to contribute to open source.

This import module is the result of many hours of analyzing CSV, and especially XML, files from a range of suppliers. I'm constantly improving the module when I discover new weird file formats. For this, I charge a sum that is small compared to the hours saved by the owner of a webshop based on dropshipping.

Notice that this sub-forum is called "Job offers and paid services". If you don't like the concept of somebody actually paying for a service you probably shouldn't come here.

/Kjeld

  • Like 1

Share this post


Link to post
Share on other sites

I can highly recommend this module.

I import over 1600 products with images under 30 seconds with a lot of features and in three different languages.

Kjeld have also give me great support! =)

Share this post


Link to post
Share on other sites

Timothy,

the job Presta-dyr is doing is awsome.
I have over 2 extreme large xml files.
one with product info
the other with product features.
I do pay more because what I asked him is muuuuuuuuch more than he had.
I'm very glad he wanted to help me.

he does the total import in 88 seconds for everything.(over 10000 articles, with mulitple pictures, and features, stock quantities, large description information,......)

I think we will be finished in this and max 2 weeks (depending on his time :-) ).
but was very negative over the xml import stuff, but what Presta-dyr did is a perfect job.
And, he is communicating, so that's allready one thing, and it works like a charm.

I think they must stick a post for this xml importer.
If you want to do business with large quantity of products, you can't be working night and day to update those products, is not possible.

The moment the total shop is online (it is now with a very small quantity of articles) I let it know.

Share this post


Link to post
Share on other sites

Hi Kjeld

I'm currently working with a demo feed for what could potentially be a massive import - if I can get it working satisfactorily.

The thing I'm interested in with this module is the facility to do the image processing. The csv I have hotlinks the images from the icecat.biz feed - the image url's link through t really good images, but of course fro prestashop we need 9 (i think) different sizes/dimensions.

How does you module deal with this - i read that it is a cronjob.... how does this work, is it controlled from the prestashop admin area?

Cheers

Blake

Share this post


Link to post
Share on other sites

Hi Blake.

Yes, the images are resized first time they are requested from the front office or via a cronjob. If your hosting service does not offer cronjobs you can use one of the free services available on the Internet. Typically it takes 1 second to resize an image.

/Kjeld

Share this post


Link to post
Share on other sites

Hmm - I'm trying to get the Icecat datafeed into prestashop.

I'm thinking in my head that if prestashop autmatically makes nine images for every item, then if I try to get hundreds of thousand images into the shop, then there are going to be huge server load implications and a massive amount of storage space needed!

Is this kind of thing even possible with prestashop?

Cheers

Blake

Share this post


Link to post
Share on other sites

Yes it's quite challenging for PrestaShop if you have > 100,000 images. A directory with thousands of files may be a bit ineffective. I know some shops (PrestaShops) have their images divided into several sub directories.

How many products do you have?

/Kjeld

Share this post


Link to post
Share on other sites

well - this is yet to be decided... the feed could potentially have 850'000 uniques - I struggle to see how this could be done realistically with prestashop.

Even if I persuaded my client to go for 200'000, that would need pretty awesome server resources. that would mean generating 1.8 million images, which would take 20+ gb of hosted images?

Share this post


Link to post
Share on other sites

yeah icecat is vast, but that is the max available, so i doubt that we would use anything like that amount.

I didn't realise that prestashop was so resource hungry in this sense though.

If I had to make a guess, I would say that with all parameters applied, the feed wouldn't exceed 50-100 thousand which is a bit more like it.

I recently tried running the kleversoft icecat module for a couple of weeks on a webfusion vps and it brought our server down. I see now that the issue is the amount of images that prestashop generates.

Share this post


Link to post
Share on other sites

  • 3 weeks later...
I offer a module that can import XML (and CSV) files. I have been working with files that import thousands of products.

One of the big problems when importing is that resizing the images takes way too long time. Typically the server can resize only 1 image per second. I have found a way to handle this.

Features
* Fast import of products from CSV og XML files.
* Support for multiple languages.
* Support for multiple images.
* Support for attributes.
* Support for features.
* ISO-8859-1 or UTF-8.
* Possible to automatically calculate the sale price based on wholesale price.
* Possible to specify delimiter for categories and sub-categories.
* Option that allows alphabetically grouping of sub-categories (useful if there are a lot of them).
* Installs as a tab under "Tools" in the back office.
* Possible to resume the import if the maximum run time is exceeded.
* Automatically creates categories and sub-categories.
* Setup for mapping of CSV or XML fields are saved in the database.
* Images are fetched and resized on demand.
* A cron job can handle the resizing of the images at nights.

Notes
* All products, manufacturers and categories are deleted when importing.
* Images are deleted from the database but never removed from the server.
* It is possible to delete all images from the back office.

If you have a lot of categories, PrestaShop tends to load slowly. I have posted a fix for this.

The price is $100.


/Kjeld


if it is possible, can you send me demo account please..

regards...

Share this post


Link to post
Share on other sites

Can you send me a demo and what it includes?

I might be interested in buying this or paying someone to do this for me on my site.

I am in desperate need to get my products imported today. I do not have the photos for the products yet but I have a spreadsheet of 8000 items that need to be imported ASAP.

If anyone is interested in helping me with that please email me at [email protected] or call me at 517-938-8404

Share this post


Link to post
Share on other sites

  • 2 weeks later...

Maybe this is what I'm looking for.

My supplier updates their prices daily, and I need to update my prices based on the wholesale prices.

I have categories/subcategories/products added already. Is your module suitable for me to update these products without deleting?

If you think it would work for me, I would like to try a demo. Please let me know what you think.

Thank you.

Share this post


Link to post
Share on other sites

  • 2 weeks later...

Hi!!

I am very interested in the module.

I have a question: Can I upgrade the stock from a listing of a provider in XML as attested by the reference number of the product?

I would like to happen to me a demonstration to proceed with the purchase.

Thanks!!

Share this post


Link to post
Share on other sites

well guys,

Presta-dyr is working on the module for me.
saw the first tests...this is it !!!!
and for that price you can not do it better.

so I would say, don't hesitate.


So the update function is working then? I'm also intrested in this for a dropship wholesale in the usa, so we can have a usa shop and only have to load their feed, and reload it every day to get the actual stock status.
When we get an order, we only have to login to their system and place a dropshipment order to the clients address. :-)

Marco

Share this post


Link to post
Share on other sites

Hi Kjeld,

First of all good job on this great module. I got the module to successfully import products from an xml feed in your demo store. However I cannot get the attributes to import. Do you know if the module supports importation of attributes or is it currently still in development?

Regards,

Geoff

Share this post


Link to post
Share on other sites

  • 2 weeks later...

Hi

I need to import some xml files to my store. Each warehouse has 500 to 10,000 products that I must import. Does your module allows you to add products to an existing category tree? Whether the module imports the attributes and several photos into one product?

I want to test this module if you can provide me a demo ?
My e-mail: [email protected]

Best regards
RamzesX

Share this post


Link to post
Share on other sites

If anyone is thinking of buying this module then don't hesitate. It is absolutely brilliant and Kjeld's support is second to none. He even helped me customise the module specifically my suppliers feed and also implement the cron jobs to run the feed.

Cheers,
Geoff

Share this post


Link to post
Share on other sites

I offer a module that can import XML (and CSV) files. I have been working with files that import thousands of products.

One of the big problems when importing is that resizing the images takes way too long time. Typically the server can resize only 1 image per second. I have found a way to handle this.

Features
* Fast import of products from CSV og XML files.
* Support for multiple languages.
* Support for multiple images.
* Support for attributes.
* Support for features.
* ISO-8859-1 or UTF-8.
* Possible to automatically calculate the sale price based on wholesale price.
* Possible to specify delimiter for categories and sub-categories.
* Option that allows alphabetically grouping of sub-categories (useful if there are a lot of them).
* Installs as a tab under "Tools" in the back office.
* Possible to resume the import if the maximum run time is exceeded.
* Automatically creates categories and sub-categories.
* Setup for mapping of CSV or XML fields are saved in the database.
* Images are fetched and resized on demand.
* A cron job can handle the resizing of the images at nights.

Notes
* All products, manufacturers and categories are optionally deleted when importing.
* Images are deleted from the database but never removed from the server.
* It is possible to delete all images from the back office.

If you have a lot of categories, PrestaShop tends to load slowly. I have posted a fix for this.

The price is $100.


/Kjeld


Hi,

I too am very interested in this module. Could you also send me a demo just to see if it is what I actually need.

Share this post


Link to post
Share on other sites

Hi ,

I bought this module too and frankly I do not regret it.

Kjled has made some modifications for me and the result is almost perfect ( because nothing is perfect :D )

I can import over 20 000 products with images, description, features in almost 90 seconds.

And at least but not last, Kjeld give me a very good support !

Chris

Share this post


Link to post
Share on other sites

Hey there

Is it possible to have access to a demo too please?

I'm looking to initially import 3,396 items in to my ps store, it'll also need to automatically update the price at time of import and the categories etc.

Also need a script to regularly interrogate my suppliers xml feed to check for stock levels (multiple times a day) and update the site with if it's in stock or not.

Can you help with this?

Thanks!

Share this post


Link to post
Share on other sites

I have updated the module with a bunch of features. One the main changes is the support for multiple suppliers. Each supplier has its own setup saved in the database. Also, I have added some nice features like:
* Deactivate categories with no active products.
* Deactivate products in deactivated categories.
* Delete all images from supplier.
* Delete all products from supplier.
* Delete all empty categories.
* Delete all products, categories and images (with selectable tables).

/Kjeld

Share this post


Link to post
Share on other sites

  • 2 weeks later...

for all those that have question..

I can say that the fast import works like a charm.
Kjeld is a good listener of needs.
So he is changing the importer time by time with new features are wanted features.

I bought his standard module and payed for some extra modifactions (like profit % marges deppending on cost) and this works awsome.
I import every day over 12.000 product, with pictures, a lot of features (over thousand), different sell prices, enormeous cateogries levels, no problem at all.

So even when prestashop is for free, this module is every euro/dollar worth....trust me on that.

Share this post


Link to post
Share on other sites

  • 2 weeks later...
  • 2 weeks later...