Jump to content
presta-dyr

[MODULE] Fast CSV or XML import of thousands products in a few seconds

Recommended Posts

I offer a module that can import XML (and CSV) files. I have been working with files that import thousands of products.

One of the big problems when importing is that resizing the images takes way too long time. Typically the server can resize only 1 image per second. I have found a way to handle this.

Features
* Fast import of products from CSV og XML files.
* Support for multiple languages.
* Support for multiple images.
* Support for multiple suppliers
* Support for attributes.
* Support for features.
* ISO-8859-1 or UTF-8.
* Possible to automatically calculate the sale price based on wholesale price.
* Possible to specify delimiter for categories and sub-categories.
* Option that allows alphabetically grouping of sub-categories (useful if there are a lot of them).
* Installs as a tab under "Tools" in the back office.
* Possible to resume the import if the maximum run time is exceeded.
* Automatically creates categories and sub-categories.
* Setup for mapping of CSV or XML fields are saved in the database.
* Images are fetched and resized on demand.
* A cron job can handle the resizing of the images at nights.
* Comes with manual in PDF format.
* Possible to update existing products.

Notes
* Products, categories etc. should not be deleted but you may automatically deactivate products that are no longer in the CSV/XML file you get from your supplier.
* It is possible to delete all images from a specific supplier from the back office.
* It is possible to delete all products from a specific supplier from the back office.
* It is possible to delete all empty categories.
* It is possible to delete all products, categories etc. if you really want to start all over.

If you have a lot of categories, PrestaShop (1.2 and 1.3) tends to load slowly. I have posted a fix for this.

The price is $100. You can buy the module from my shop.

I will no longer offer a demo (I spent way too much time managing my demo site). Instead I have added some demonstration videos (look in the bottom of the description).


Edit 2010-10-20: Updated with new features like multi-supplier.
Edit 2011-05-21: Works with PrestaShop 1.4.
Edit 2012-08-05: Works with PrestaShop 1.5.
Edit 2012-11-02: Added link to shop.

Edit 2015-05-06: Works with PrestaShop 1.6.

/Kjeld

Edited by presta-dyr (see edit history)
  • Like 9

Share this post


Link to post
Share on other sites


Notes
* All products, manufacturers and categories are deleted when importing.

The price is $100.
/Kjeld


Nice module, but what about if someone needs only to import products into existing categories?

Share this post


Link to post
Share on other sites

Actually, there is an option for exactly that (i.e. preserve categories).

Cheers,
Kjeld

Share this post


Link to post
Share on other sites

Hi,
Can you send a demo to robertino.m@gmail.com so I can be sure it's working ?
I'm using Presta 1.3 btw.
Thanks

Share this post


Link to post
Share on other sites

Id like to see a demo if possible also - do you also have a sample cvs file for me to review..

Share this post


Link to post
Share on other sites

Sure, I have a bunch of (authentic) CSV and XML files I have used for test. I guess you are looking for something specific? I mean a simple, short CSV file works fine with the existing import function.

/Kjeld

Share this post


Link to post
Share on other sites

I have sent you a mail on your .dk domain but it was rejected by server.
Robert

Share this post


Link to post
Share on other sites

No, not yet, something must be wrong. Email is robertino.m@gmail.com ?

Share this post


Link to post
Share on other sites
Actually, there is an option for exactly that (i.e. preserve categories).

Cheers,
Kjeld


Is there an option so that you can only add products?
I have my categories and I have 500 products which I need to keep them.

Share this post


Link to post
Share on other sites

Currently it is not possible to add products to an existing base. It is much faster just to add products from scratch than to update products. But I have an idea how it can be made...

/Kjeld

Share this post


Link to post
Share on other sites

Presta-dyr

look at my post : http://www.prestashop.com/forums/viewthread/58071/job_offers_and_paid_services/i_need_a_xml_importer

two files (one xml example, other is a field setting document)

can your importer do this?
ps : product can NOT be deleted when starting import.
only when the status code <>"NORMAL" then you can delete a product.

this because I have two suppliers.
one small (over 450 articles) I've added manualy those products.
the other over 15000 articles I must update/delete/add every week/night.

look at my post and I'll hear it back from you.

Share this post


Link to post
Share on other sites

Hi deech123.

Currently the module does not support appending or updating products. But if you're willing to pay for it I can make a special version that suits your needs.

I have sent you an e-mail.


/Kjeld

Share this post


Link to post
Share on other sites

I'll be interested too in this module supporting appending and updates; so far i haven't had the time to fully test this module due to the countless problems I have after upgrading presta to last version; I hope that I'll finish this in a few days and get back to testing because god knows I'm in great need of a such working module.

Share this post


Link to post
Share on other sites

where are the open source submitters..? lets get on it for the benefit of all!!!

Share this post


Link to post
Share on other sites

I have submitted a number of modules (for free) and have spent a lot of times translating PrestaShop to Danish. I do my part to contribute to open source.

This import module is the result of many hours of analyzing CSV, and especially XML, files from a range of suppliers. I'm constantly improving the module when I discover new weird file formats. For this, I charge a sum that is small compared to the hours saved by the owner of a webshop based on dropshipping.

Notice that this sub-forum is called "Job offers and paid services". If you don't like the concept of somebody actually paying for a service you probably shouldn't come here.

/Kjeld

  • Like 1

Share this post


Link to post
Share on other sites

Show us it can be done successfully and correctly - Demo would be nice.. with example cvs.. and categories. thx

Share this post


Link to post
Share on other sites

I can highly recommend this module.

I import over 1600 products with images under 30 seconds with a lot of features and in three different languages.

Kjeld have also give me great support! =)

Share this post


Link to post
Share on other sites

well guys,

Presta-dyr is working on the module for me.
saw the first tests...this is it !!!!
and for that price you can not do it better.

so I would say, don't hesitate.

Share this post


Link to post
Share on other sites

hi,

I also need this module presta-dyr to update existing + add new ...

so you already have 2 clients for that 'deech123 and me'

Share this post


Link to post
Share on other sites

Timothy,

the job Presta-dyr is doing is awsome.
I have over 2 extreme large xml files.
one with product info
the other with product features.
I do pay more because what I asked him is muuuuuuuuch more than he had.
I'm very glad he wanted to help me.

he does the total import in 88 seconds for everything.(over 10000 articles, with mulitple pictures, and features, stock quantities, large description information,......)

I think we will be finished in this and max 2 weeks (depending on his time :-) ).
but was very negative over the xml import stuff, but what Presta-dyr did is a perfect job.
And, he is communicating, so that's allready one thing, and it works like a charm.

I think they must stick a post for this xml importer.
If you want to do business with large quantity of products, you can't be working night and day to update those products, is not possible.

The moment the total shop is online (it is now with a very small quantity of articles) I let it know.

Share this post


Link to post
Share on other sites

deech123,

i'm waiting to see the result ...

(i saw you've a belgian shop (me also))

updating is hell ...

Share this post


Link to post
Share on other sites

Hi Timothy.

I actually have implemented the possibility to keep the existing products/manufacturers/features/categories etc.

Send me an e-mail if you want a demo.

/Kjeld

Share this post


Link to post
Share on other sites

Hi Kjeld

I'm currently working with a demo feed for what could potentially be a massive import - if I can get it working satisfactorily.

The thing I'm interested in with this module is the facility to do the image processing. The csv I have hotlinks the images from the icecat.biz feed - the image url's link through t really good images, but of course fro prestashop we need 9 (i think) different sizes/dimensions.

How does you module deal with this - i read that it is a cronjob.... how does this work, is it controlled from the prestashop admin area?

Cheers

Blake

Share this post


Link to post
Share on other sites

Hi Blake.

Yes, the images are resized first time they are requested from the front office or via a cronjob. If your hosting service does not offer cronjobs you can use one of the free services available on the Internet. Typically it takes 1 second to resize an image.

/Kjeld

Share this post


Link to post
Share on other sites

ah right - so I make a cronjob to execute a php file that you provide with the module that generates the images?

please excuse my ignorance, I have never used cron before!

Blake

Share this post


Link to post
Share on other sites

Hmm - I'm trying to get the Icecat datafeed into prestashop.

I'm thinking in my head that if prestashop autmatically makes nine images for every item, then if I try to get hundreds of thousand images into the shop, then there are going to be huge server load implications and a massive amount of storage space needed!

Is this kind of thing even possible with prestashop?

Cheers

Blake

Share this post


Link to post
Share on other sites

Yes it's quite challenging for PrestaShop if you have > 100,000 images. A directory with thousands of files may be a bit ineffective. I know some shops (PrestaShops) have their images divided into several sub directories.

How many products do you have?

/Kjeld

Share this post


Link to post
Share on other sites

well - this is yet to be decided... the feed could potentially have 850'000 uniques - I struggle to see how this could be done realistically with prestashop.

Even if I persuaded my client to go for 200'000, that would need pretty awesome server resources. that would mean generating 1.8 million images, which would take 20+ gb of hosted images?

Share this post


Link to post
Share on other sites

Nearly a million products! That sounds challenging. I wonder how many items there are in the biggest PrestaShop available?

/Kjeld

Share this post


Link to post
Share on other sites

yeah icecat is vast, but that is the max available, so i doubt that we would use anything like that amount.

I didn't realise that prestashop was so resource hungry in this sense though.

If I had to make a guess, I would say that with all parameters applied, the feed wouldn't exceed 50-100 thousand which is a bit more like it.

I recently tried running the kleversoft icecat module for a couple of weeks on a webfusion vps and it brought our server down. I see now that the issue is the amount of images that prestashop generates.

Share this post


Link to post
Share on other sites
Guest

How to import of Size Austribute and groups in a few seconds ?

Share this post


Link to post
Share on other sites

I made support for attributes. But I'm revising it, so currently it's not in the code. Do you have an example?

/Kjeld

Share this post


Link to post
Share on other sites
I offer a module that can import XML (and CSV) files. I have been working with files that import thousands of products.

One of the big problems when importing is that resizing the images takes way too long time. Typically the server can resize only 1 image per second. I have found a way to handle this.

Features
* Fast import of products from CSV og XML files.
* Support for multiple languages.
* Support for multiple images.
* Support for attributes.
* Support for features.
* ISO-8859-1 or UTF-8.
* Possible to automatically calculate the sale price based on wholesale price.
* Possible to specify delimiter for categories and sub-categories.
* Option that allows alphabetically grouping of sub-categories (useful if there are a lot of them).
* Installs as a tab under "Tools" in the back office.
* Possible to resume the import if the maximum run time is exceeded.
* Automatically creates categories and sub-categories.
* Setup for mapping of CSV or XML fields are saved in the database.
* Images are fetched and resized on demand.
* A cron job can handle the resizing of the images at nights.

Notes
* All products, manufacturers and categories are deleted when importing.
* Images are deleted from the database but never removed from the server.
* It is possible to delete all images from the back office.

If you have a lot of categories, PrestaShop tends to load slowly. I have posted a fix for this.

The price is $100.


/Kjeld


if it is possible, can you send me demo account please..

regards...

Share this post


Link to post
Share on other sites

Can you send me a demo and what it includes?

I might be interested in buying this or paying someone to do this for me on my site.

I am in desperate need to get my products imported today. I do not have the photos for the products yet but I have a spreadsheet of 8000 items that need to be imported ASAP.

If anyone is interested in helping me with that please email me at charbennett@gmail.com or call me at 517-938-8404

Share this post


Link to post
Share on other sites

Maybe this is what I'm looking for.

My supplier updates their prices daily, and I need to update my prices based on the wholesale prices.

I have categories/subcategories/products added already. Is your module suitable for me to update these products without deleting?

If you think it would work for me, I would like to try a demo. Please let me know what you think.

Thank you.

Share this post


Link to post
Share on other sites

Hi,

I am thinking of switching to prestashop but need this module too.. Does it import any xml feed? Does it map the feed automatically to work with prestashop?

Could i test the demo please?

Thanks

Share this post


Link to post
Share on other sites

Hi!!

I am very interested in the module.

I have a question: Can I upgrade the stock from a listing of a provider in XML as attested by the reference number of the product?

I would like to happen to me a demonstration to proceed with the purchase.

Thanks!!

Share this post


Link to post
Share on other sites

Hi,

I too am very interested in this module. Could you also send me a demo just to see if it is what I actually need.

Cheers,

Geoff

Share this post


Link to post
Share on other sites
well guys,

Presta-dyr is working on the module for me.
saw the first tests...this is it !!!!
and for that price you can not do it better.

so I would say, don't hesitate.


So the update function is working then? I'm also intrested in this for a dropship wholesale in the usa, so we can have a usa shop and only have to load their feed, and reload it every day to get the actual stock status.
When we get an order, we only have to login to their system and place a dropshipment order to the clients address. :-)

Marco

Share this post


Link to post
Share on other sites

Hi Kjeld,

First of all good job on this great module. I got the module to successfully import products from an xml feed in your demo store. However I cannot get the attributes to import. Do you know if the module supports importation of attributes or is it currently still in development?

Regards,

Geoff

Share this post


Link to post
Share on other sites

Hi,

I too am very interested in this module. Could you also send me a demo just to see if it is what I actually need.

Ciao ;-)

Share this post


Link to post
Share on other sites

Hi

I need to import some xml files to my store. Each warehouse has 500 to 10,000 products that I must import. Does your module allows you to add products to an existing category tree? Whether the module imports the attributes and several photos into one product?

I want to test this module if you can provide me a demo ?
My e-mail: biuro@giftforyou.pl

Best regards
RamzesX

Share this post


Link to post
Share on other sites

Hi there,
Sounds just what i am looking for?
Please can you PM or email me over a demo with sample data?
john@car-vision.co.uk

I too would like to test before commiting to see if it will work in the way we need.

Thanks in advance,

John.

Share this post


Link to post
Share on other sites

If anyone is thinking of buying this module then don't hesitate. It is absolutely brilliant and Kjeld's support is second to none. He even helped me customise the module specifically my suppliers feed and also implement the cron jobs to run the feed.

Cheers,
Geoff

Share this post


Link to post
Share on other sites
Guest danielias

Can you send me a demo too?

danielsantana@email.com

Thank you

Share this post


Link to post
Share on other sites
I offer a module that can import XML (and CSV) files. I have been working with files that import thousands of products.

One of the big problems when importing is that resizing the images takes way too long time. Typically the server can resize only 1 image per second. I have found a way to handle this.

Features
* Fast import of products from CSV og XML files.
* Support for multiple languages.
* Support for multiple images.
* Support for attributes.
* Support for features.
* ISO-8859-1 or UTF-8.
* Possible to automatically calculate the sale price based on wholesale price.
* Possible to specify delimiter for categories and sub-categories.
* Option that allows alphabetically grouping of sub-categories (useful if there are a lot of them).
* Installs as a tab under "Tools" in the back office.
* Possible to resume the import if the maximum run time is exceeded.
* Automatically creates categories and sub-categories.
* Setup for mapping of CSV or XML fields are saved in the database.
* Images are fetched and resized on demand.
* A cron job can handle the resizing of the images at nights.

Notes
* All products, manufacturers and categories are optionally deleted when importing.
* Images are deleted from the database but never removed from the server.
* It is possible to delete all images from the back office.

If you have a lot of categories, PrestaShop tends to load slowly. I have posted a fix for this.

The price is $100.


/Kjeld


Hi,

I too am very interested in this module. Could you also send me a demo just to see if it is what I actually need.

Share this post


Link to post
Share on other sites

Sounds interesting... if i can update mine database with new products then this is module for me.
PM me if you have some demo site to test it.
Cheers!

Share this post


Link to post
Share on other sites

One more question... can i upload attributes... periodicaly?
Theres a bug so i can't use BO to upload attributes becouse Presta 1.1 simply duplicates them. :(

Share this post


Link to post
Share on other sites

Hi ,

I bought this module too and frankly I do not regret it.

Kjled has made some modifications for me and the result is almost perfect ( because nothing is perfect :D )

I can import over 20 000 products with images, description, features in almost 90 seconds.

And at least but not last, Kjeld give me a very good support !

Chris

Share this post


Link to post
Share on other sites

Hey there

Is it possible to have access to a demo too please?

I'm looking to initially import 3,396 items in to my ps store, it'll also need to automatically update the price at time of import and the categories etc.

Also need a script to regularly interrogate my suppliers xml feed to check for stock levels (multiple times a day) and update the site with if it's in stock or not.

Can you help with this?

Thanks!

Share this post


Link to post
Share on other sites

Hi,

please could you allow me to try your module demo?
We're really interested in to your module, looks to be perfect for us!
Thanks!

Share this post


Link to post
Share on other sites

I am very much interested in this module as well. I am going to be using a constant XML service to add/update/delete products. I would love to try a demo.

Share this post


Link to post
Share on other sites

I have updated the module with a bunch of features. One the main changes is the support for multiple suppliers. Each supplier has its own setup saved in the database. Also, I have added some nice features like:
* Deactivate categories with no active products.
* Deactivate products in deactivated categories.
* Delete all images from supplier.
* Delete all products from supplier.
* Delete all empty categories.
* Delete all products, categories and images (with selectable tables).

/Kjeld

Share this post


Link to post
Share on other sites

Hi Kjeld,

Can you send me a demo and tell me how to buy the module - this sounds like it will solve our import problems.

Thanks!

Share this post


Link to post
Share on other sites

for all those that have question..

I can say that the fast import works like a charm.
Kjeld is a good listener of needs.
So he is changing the importer time by time with new features are wanted features.

I bought his standard module and payed for some extra modifactions (like profit % marges deppending on cost) and this works awsome.
I import every day over 12.000 product, with pictures, a lot of features (over thousand), different sell prices, enormeous cateogries levels, no problem at all.

So even when prestashop is for free, this module is every euro/dollar worth....trust me on that.

Share this post


Link to post
Share on other sites

I'm interested in this module... I already sent you an e-mail... did you receive it presta-dyr ?

Edit : Sorry... I just find your response in my inbox :)

Share this post


Link to post
Share on other sites
Please send me an e-mail for more info.

/Kjeld


Hi

I’m interesting for prchase this module.
Can you send me more details or a demo link?

Thank you.

Carlo Alberto.

Share this post


Link to post
Share on other sites

Hi,
I have sent you an email. I know the basic prestashop should import images but that's what i need and can't manage to do that. Hope i can try a demo.

Share this post


Link to post
Share on other sites

Hi,

I am looking for an XML import module for Prestashop for a new site I am working on. Can you please email me at charlesa69@hotmail.com.

Charles

Share this post


Link to post
Share on other sites

Hi,

I'm interested in buying your mod please email me at rnyamek@yahoo.com.

Thanks,
Richmond

Share this post


Link to post
Share on other sites

Hi Task001,

I can't show examples with XML/CSV files from actual suppliers because they are normally confidential. I plan writing a manual in the near future, so I will probably have to make up my own XML/CSV files so I can show some examples.

/Kjeld

Share this post


Link to post
Share on other sites

Hi Jjeld

I’m interested in buying your mod and would like a demo if possible please email me at wayneensor@techideal.co.uk.

Best regards

Wayne

Share this post


Link to post
Share on other sites

hi, can you send me a demo license of the module??? i need to know if this plugin works for me or not before purchase it.

Thank you very much.

Share this post


Link to post
Share on other sites