Copy Shopdata: Script for upgrading and fixing buggy shops by copying business data and images

[a-d-g]

On 12/28/2015 at 2:11 PM, Rudolfo78 said:

Hi musicmaster,

I'm trying script copy_shopdata on my localhost installations - both 1.6.0.9 and I got stuck at the very beginning of copy process

 

1) old installation - I've downloaded my current installation 1.6.0.9 via FTP & exported database.sql and configured it to run localy (OS X, MAMP, Sequel Pro).

 

2) new installation - I've installed fresh installation same version 1.6.0.9 on my localhost, where I added subfolder "triple_edit" with its files, as well as script copy_shopdata files

13:45:32
1 old languages; 1 new languages. Language check ok.


accessory 1123 1
MySQL error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-new.ps_accessory SELECT `id_product_1`,`id_product_2` FROM maclife-old.ps_acces' at line 1
Generated by Query 'INSERT INTO maclife-new.ps_accessory SELECT `id_product_1`,`id_product_2` FROM maclife-old.ps_accessory'

I've checked manual but I didn't find any useful information.

 

Could you please give me some advice, what should I check or revise. As I'm only basic user with some additional skills, I'm not very good in coding, syntax, etc.

 

Thanks for any advice.

 

On 12/28/2015 at 2:57 PM, musicmaster said:

I am not totally sure. But my guess is that the hyphens ("-") in the database names are the problem. They might be interpreted as "minus". The solution is to put the server names between backticks.I will have a look at it.

I have found new scenery where that error code happen, for example in this query:

SELECT d_specific_price, id_specific_price_rule, id_cart, id_product, id_shop, id_shop_group, id_currency, id_country, id_group, id_customer, id_product_attribute, price,from_quantity,reduction,reduction_tax,reduction_type,from,to INTO OUTFILE '/var/xxxxxxxxxxxxxxxxxxx/ptools/copy_shopdata_specific_price.dtx' FROM `ps_specific_price`

In this case error cause is the name of last fields of table "from" and "to", this field names are used in mysql querys, also sqlserver undertand it is part of query, not fields. the solution is close inside braquets  SELECT ........,'from','to' INTO OUTFILE...

 

 

Edited April 2, 2020 by salvavidal (see edit history)

[musicmaster]

9 hours ago, salvavidal said:

 

I have found new scenery where that error code happen, for example in this query:

SELECT d_specific_price, id_specific_price_rule, id_cart, id_product, id_shop, id_shop_group, id_currency, id_country, id_group, id_customer, id_product_attribute, price,from_quantity,reduction,reduction_tax,reduction_type,from,to INTO OUTFILE '/var/xxxxxxxxxxxxxxxxxxx/ptools/copy_shopdata_specific_price.dtx' FROM `ps_specific_price`

In this case error cause is the name of last fields of table "from" and "to", this field names are used in mysql querys, also sqlserver undertand it is part of query, not fields. the solution is close inside braquets  SELECT ........,'from','to' INTO OUTFILE...

 

 

Hi,

I have uploaded a new version. Can you try that?

 

[a-d-g]

1 hour ago, musicmaster said:

Hi,

I have uploaded a new version. Can you try that?

 

I have detected other issue, in the finish log the script write all tables that has not been copied from old shop, you remove automaticly first letters of table to remove PREFIX, but in case of my old shop had tables without PREFIX, those will appear cutted, for example, i have a table called 'updates', is a table which i created to process an internal script, that table appear like 'ates' on final log, Do you understand?, i guess you had to compare PREFIX of shop with begining of name and dont remove 3 first letter in case it doesnt match with PREFIX.

 

Than you for your big work, i will try your new version.

Edited April 2, 2020 by salvavidal (see edit history)

[a-d-g]

1 hour ago, musicmaster said:

Hi,

I have uploaded a new version. Can you try that?

 

I have not tried because i don't need to do, i have checked code in copy_shopdata_functions.inc.php, and you have solved right with this line:

$args = "`".implode("`,`", array_intersect($newfields, $oldfields))."`";

Good job, remember to solve the other issue about cutting names which has not PREFIX in old shop, in screen appear incomplete the name.

[musicmaster]

1 hour ago, salvavidal said:

I have detected other issue, in the finish log the script write all tables that has not been copied from old shop, you remove automaticly first letters of table to remove PREFIX, but in case of my old shop had tables without PREFIX, those will appear cutted, for example, i have a table called 'updates', is a table which i created to process an internal script, that table appear like 'ates' on final log, Do you understand?, i guess you had to compare PREFIX of shop with begining of name and dont remove 3 first letter in case it doesnt match with PREFIX.

This point is not clear enough to me. The old and the new prefix are removed as they may be different. But that applies to all tables not just those not copied. And I don't see any cutting in the code that writes the "$oldextras" (lines 170-175 in copy_shopdata_functions.inc.php).

[musicmaster]

23 hours ago, salvavidal said:

I have detected other issue, in the finish log the script write all tables that has not been copied from old shop, you remove automaticly first letters of table to remove PREFIX, but in case of my old shop had tables without PREFIX, those will appear cutted, for example, i have a table called 'updates', is a table which i created to process an internal script, that table appear like 'ates' on final log, Do you understand?, i guess you had to compare PREFIX of shop with begining of name and dont remove 3 first letter in case it doesnt match with PREFIX.

Maybe this file solves that problem?

copy_shopdata.php

[a-d-g]

11 hours ago, musicmaster said:

Maybe this file solves that problem?

copy_shopdata.php 11.07 kB · 0 downloads

it doesn't solve, and in my opinion is not a good solution, your are removing tables without PREFIX in your log printed when script finish, in my case i want to know which table weren't copied, even if those tables dont have PREFIX, because were tables created to work with my personal scripts, and i need to use it in new database, is good to show it in "not copied warning".

OLD TABLES						NOT COPIED (this way is shown when script finish)
actualizar_stock					ualizar_stock
tabla_auxiliar						la_auxiliar
ps_example1						example1
ps_example2						example2
ps_example3						example3
...								...
...								...

As you can see above those table in old prestashop which doesn't have PREFIX will be shown cutted on final log, you can fix it with next change in line 36:

while($row = mysqli_fetch_row($res))
{ $oldtables[] = substr($row[0], $len);
}

you have to edit with this:

while($row = mysqli_fetch_row($res))
{ $oldtables[] = substr($row[0], 0, $len) == _OLD_PREFIX_  ? substr($row[0], $len) : $row[0];
}

Thanks.

Edited April 4, 2020 by salvavidal (see edit history)

[musicmaster]

Is this solution better? It lists the tables that are not considered.

Prefixes are used to have several shops in one database. That is useful for shops whose hosting provider only supplies one database. So the lack of a filter was a kind of bug.

copy_shopdata.php

[a-d-g]

1 hour ago, musicmaster said:

Is this solution better? It lists the tables that are not considered.

Prefixes are used to have several shops in one database. That is useful for shops whose hosting provider only supplies one database. So the lack of a filter was a kind of bug.

copy_shopdata.php 11.38 kB · 0 downloads

Ok, you are right, in case that you have one database for several shops, my change is not good idea, i have checked you new update, and i see you have created a new array with "wrong prefix tables", also as you have said, if you have other shops with other prefix, the tables of all shops will be put inside of this array, because you only compare with _OLD_PREFIX_,

[a-d-g]

I have found another important 'data' which have to be copied to new shop, the following values from ps_configuration:

PS_CUSTOMER_GROUP

PS_UNIDENTIFIED_GROUP

PS_GUEST_GROUP

It is very important to copy those values, that way you could avoid several errors is you have edited customer groups in original shop, if you have never edited customer group, you dont need to copy those values.

 

[musicmaster]

1 hour ago, salvavidal said:

I have found another important 'data' which have to be copied to new shop, the following values from ps_configuration:

PS_CUSTOMER_GROUP

PS_UNIDENTIFIED_GROUP

PS_GUEST_GROUP

It is very important to copy those values, that way you could avoid several errors is you have edited customer groups in original shop, if you have never edited customer group, you dont need to copy those values.

 

Thank you for the addition. I have added them in a new version of the config file.

[a-d-g]

a new question, if fields in old_table are in different position than fields in new_table, does script copy tables in right position? I have seen some error in tables because fields of old an new table doesnt have same positions.

[musicmaster]

Just now, salvavidal said:

a new question, if fields in old_table are in different position than fields in new_table, does script copy tables in right position? I have seen some error in tables because fields of old an new table doesnt have same positions.

Do you mean that in the old table the order is id-name and in the new one name-id? Yes, that and missing fields in one of the tables are handled correctly.

One thing that can cause problems is when the new table has a unique key that in the old one wasn't unique.

[a-d-g]

9 minutes ago, musicmaster said:

Do you mean that in the old table the order is id-name and in the new one name-id? Yes, that and missing fields in one of the tables are handled correctly.

One thing that can cause problems is when the new table has a unique key that in the old one wasn't unique.

not, i mean when oldtable has fields

id_name,

field2,

field3,

field4

and newtable has fields

field2,

id_name,

field3,

field4

as you can see, the same fields but different position.

 

Edited April 10, 2020 by salvavidal (see edit history)

[musicmaster]

5 hours ago, salvavidal said:

not, i mean when oldtable has fields

id_name,

field2,

field3,

field4

and newtable has fields

field2,

id_name,

field3,

field4

as you can see, the same fields but different position.

 

I could have been clearer but that was what I meant. It is handled correctly.

[rcmp_presta]

Unfortunalelly the script doesn't work for me.

[musicmaster]

15 hours ago, rcmp_presta said:

Unfortunalelly the script doesn't work for me.

No helpdesk will be able to help you when you don't get more concrete than "It doesn't work".

[gorkij77]

Hi, might I inquire as to the progress on making this compatible to v.1.7.7 ? 

[musicmaster]

On 12/27/2020 at 4:22 PM, gorkij77 said:

Hi, might I inquire as to the progress on making this compatible to v.1.7.7 ? 

I hope to release Prestools for 1.7.7 at the end of the week. Copy_shopdata should work without changes.

[gorkij77]

9 hours ago, musicmaster said:

I hope to release Prestools for 1.7.7 at the end of the week. Copy_shopdata should work without changes.

Thank you MusicMaster, love Prestools. I did however download and try to run copy_shopdata with the most recent install and it simply said that it wasn't compatible?

[musicmaster]

I see. It uses the same login system as Prestools and the version check is in login1.php.

[gorkij77]

Quite fine either way, seems I've got a half-baked upgrade to 1.7.7 so I'll need to do a migration either way. But no hurries, thank you for your excellent work and happy new year!

[a-d-g]

Hi,

I'm trying to copy from PS 1.6.1.23 to new installed PS 1.6.1.23, but i dont know why some tables cant be copied, for example ps_employee

 

other problem is, if a table exist in old database and dont exist in new database, the script stop with this message:

 

"MySQL error 1146: Table 'update.p7_mailalert_customer_oos' doesn't exist
Generated by URL '/gestion/ptools/copy_shopdata.php'
with Query 'SHOW COLUMNS FROM p7_mailalert_customer_oos'"

 

I'm using last version of copy_shopdata.035

Edited January 12, 2021 by salvavidal
add information (see edit history)

[musicmaster]

On 1/12/2021 at 12:06 PM, salvavidal said:

Hi,

I'm trying to copy from PS 1.6.1.23 to new installed PS 1.6.1.23, but i dont know why some tables cant be copied, for example ps_employee

 

other problem is, if a table exist in old database and dont exist in new database, the script stop with this message:

 

"MySQL error 1146: Table 'update.p7_mailalert_customer_oos' doesn't exist
Generated by URL '/gestion/ptools/copy_shopdata.php'
with Query 'SHOW COLUMNS FROM p7_mailalert_customer_oos'"

 

I'm using last version of copy_shopdata.035

I recognize neither problem:

 - copying employee shouldn't be a problem. What is the problem that you see?

 - the script checks whether tables exist in the old and new database and copying only starts when they are present in both. You can always delete the name from the config file, but it just puzzles me why this isn't handled as it should be.

 

 

 

[musicmaster]

Prestashop 1.7.6.0 and higher have a new table: ps_currency_lang. 

Until version 0.36 Copy_shopdata doesn't list this table in the config file. As a consequence copying tables may result in missing currency translations. That causes a "Missing Locale" error. 

In version 0.37 ps_image_type has been removed from the list of copied tables. At closer look its presence wasn't a good idea.

In version 0.38 localhost and 127.0.0.1 are considered the same so copying happens now direct instead of via the file system. Also a bug was fixed that made copying over the file system go wrong when tables didn't have the same fields.

Version 0.39 added some fields that are copied.

 

Edited January 23, 2021 by musicmaster (see edit history)

[a-d-g]

HI, what's the last prestashop version which works well with copy_shopdata ?

 

does it work with PS1.7.7.0 ???

[musicmaster]

On 1/27/2021 at 7:44 PM, salvavidal said:

HI, what's the last prestashop version which works well with copy_shopdata ?

 

does it work with PS1.7.7.0 ???

Yes, it does

[a-d-g]

I have detected a little bug.

 

Sometimes if i try to migrate from one shop to other shop with differents languages iso codes, copyshopdata script remove languages from old_db, if I run script again, it remove the rest and let old_db without languages. i attach an image of this.

 

Edited January 29, 2021 by salvavidal (see edit history)

[musicmaster]

On 1/29/2021 at 9:26 AM, salvavidal said:

I have detected a little bug.

 

Sometimes if i try to migrate from one shop to other shop with differents languages iso codes, copyshopdata script remove languages from old_db, if I run script again, it remove the rest and let old_db without languages. i attach an image of this.

 

This is strange. Copy_shopdata does not change the old database. It just copies the specified tables to the new database and makes all the language id changes there. And neither in the old nor in the new database does it change the ps_lang table - what must have been done in order to achieve the "0 old languages"  that you see.

So my best guess would be that you made changes in the code or the database that didn't work out well. 

If you can reproduce the deletion can you run it with 'verbose="true"' and pm me the outcome? Then would enable me to check more in detail for what causes the problem you experienced. And if it was something in CS I would certainly notice it.

Edited February 2, 2021 by musicmaster (see edit history)

[a-d-g]

On 1/29/2021 at 8:58 PM, musicmaster said:

This is strange. Copy_shopdata should change nothing in the old database and I can't find any code where it does. It just copies the specified tables to the new database and makes there all the language changes.

If you didn't change anything in the code it must be that somewhere the old and the new server are exchanged. 

If you can reproduce the deletion can you run it with 'verbose="true"' and pm me the outcome?

I will tray to reproduce it, but not easy

[musicmaster]

7 hours ago, salvavidal said:

I will tray to reproduce it, but not easy

Can you at least try to describe the situation? A language id of 6 for the new shop is a bit unusual.

[dandumit]

Hi,

I am trying to run this script to migrate data from a presta 1.6.15 to 1.7.7.1

Script starts ,logs some data but after that I get message :

 

1 accessory 2204

MySQL error 1045: Access denied for user 'user'@'localhost' (using password: YES)
Generated by URL '/admin045ur9jez/preztoolz/copy_shopdata.php'
with Query 'SELECT `id_product_1`,`id_product_2` INTO OUTFILE '/home/admin/domains/asd/public_html/admin045ur9jez/preztoolz/export/copy_shopdata_accessory.dtx' FROM `ps_accessory`'

 

LE: 

I have run as root :

 GRANT FILE ON *.* TO 'user'@'localhost';

Now I get the error : 

MySQL error 1: Can't create/write to file '/../preztoolz/copy_shopdata_accessory.dtx' (Errcode: 13 "Permission denied")
 

still digging... 

 

Thank you,

Daniel

Edited February 21, 2021 by dandumit (see edit history)

[musicmaster]

On 2/21/2021 at 7:25 AM, dandumit said:

Hi,

I am trying to run this script to migrate data from a presta 1.6.15 to 1.7.7.1

Script starts ,logs some data but after that I get message :

 

1 accessory 2204

MySQL error 1045: Access denied for user 'user'@'localhost' (using password: YES)
Generated by URL '/admin045ur9jez/preztoolz/copy_shopdata.php'
with Query 'SELECT `id_product_1`,`id_product_2` INTO OUTFILE '/home/admin/domains/asd/public_html/admin045ur9jez/preztoolz/export/copy_shopdata_accessory.dtx' FROM `ps_accessory`'

LE: 

I have run as root :

 GRANT FILE ON *.* TO 'user'@'localhost';

Now I get the error : 

MySQL error 1: Can't create/write to file '/../preztoolz/copy_shopdata_accessory.dtx' (Errcode: 13 "Permission denied")

Copy_shopdata is meant for transporting data between shops of the same version. The gap between 1.6.1.5 and 1.7.7.1 will nearly certainly produce problems.

As for the rights problem, I don't know. Did you check for directory rights?

 

[dandumit]

11 hours ago, musicmaster said:

Copy_shopdata is meant for transporting data between shops of the same version. The gap between 1.6.1.5 and 1.7.7.1 will nearly certainly produce problems.

I saw here on forum a report from a user that has upgraded using your script.  I have even bought the most acclaimed module migration pro but it does not work well at all !!!

I am kind of desperate to find a solution to migrate from this.

I was looking trough the sql's from 1click migrate and didn't look to me that database structure has changed that much ...

Please tell me , is there any way to force to use migration directly and not using local files ? 

I have looked on your scripts and you where comparing both database config and based on that calling copy "on the fly".

Thank you for script !

Daniel

[musicmaster]

4 hours ago, dandumit said:

I saw here on forum a report from a user that has upgraded using your script.  I have even bought the most acclaimed module migration pro but it does not work well at all !!!

I am kind of desperate to find a solution to migrate from this.

I was looking trough the sql's from 1click migrate and didn't look to me that database structure has changed that much ...

Please tell me , is there any way to force to use migration directly and not using local files ? 

I have looked on your scripts and you where comparing both database config and based on that calling copy "on the fly".

Thank you for script !

Daniel

Did you read the manual of Copy_shopdata?

Copy_shopdata can both copy direct and via a file. It copies via file when the databases are on different servers. So if you have different servers and want direct copy you should copy the darabase to the new server (Copy_shopdata doesn't look at the files).

Copy_shopdata contains fixes for the two most deadly differences when you upgrade this way (the state field in PS 1.7 ps_product that should be 1 otherwise you don't see the products and the ps_currency_lang table - introduced in 1.7.6 - that crashes your site with a "missing locale" error when not correctly filled). But I never made a systematic study so there very likely are other problems.

Are you using the latest version? Copy_shopdata has changed a lot in the last two months.

Edited February 23, 2021 by musicmaster (see edit history)

[dandumit]

16 minutes ago, musicmaster said:

Did you read the manual of Copy_shopdata?

Copy_shopdata can both copy direct and via a file. It copies via file when the databases are on different servers. So if you have different servers and want direct copy you should copy the darabase to the new server (Copy_shopdata doesn't look at the files).

HEllo @musicmaster ! thank you for script and answers .

I have read manual and I have tried to look on code. 

I have found the issue that script compares server and username and password.  I have installed source server in subdomain at same server but a different user to avoid mixing them.

  //&& (_OLD_USER_ == _DB_USER_) && (_OLD_PASSWD_ == _DB_PASSWD_)
  if(($oldserver == $newserver) )

Please tell me do I have to erase all tables ?  I didn't seen do truncate or to insert only difference.

Thank you

DAniel

 

[musicmaster]

57 minutes ago, dandumit said:

HEllo @musicmaster ! thank you for script and answers .

I have read manual and I have tried to look on code. 

I have found the issue that script compares server and username and password.  I have installed source server in subdomain at same server but a different user to avoid mixing them.

  //&& (_OLD_USER_ == _DB_USER_) && (_OLD_PASSWD_ == _DB_PASSWD_)
  if(($oldserver == $newserver) )

Please tell me do I have to erase all tables ?  I didn't seen do truncate or to insert only difference.

Thank you

DAniel

 

No, you shouldn't erase tables.

The script truncates the tables and then inserts the new data. That way the data structure matches that of the new version and you avoid missing field errors.

[dandumit]

@musicmaster - it's my pleasure to contradict you. I have run the script and for now at least front office it seems to be working.

I cannot log in in backoffice.  

LE : 

reading the manual (RTFM  ) :

- copy the security strings (Ryndael keys, Cookie keys, etc) from the settings.inc.php of the old shop to the settings.inc.php of the new shop. Without this you won't be able to login. [Note: in Prestashop 1.7 the keys are in parameters.php but at least for the employee password this is not needed]

LE 2 : 

copied \public_html\test16\config\settings.inc.php

to 

\public_html\app\config\parameters.php

cookie values and I have been able to log in backoffice

Edited February 24, 2021 by dandumit (see edit history)

[musicmaster]

7 hours ago, dandumit said:

@musicmaster - it's my pleasure to contradict you. I have run the script and for now at least front office it seems to be working.

I cannot log in in backoffice.  

LE : 

reading the manual (RTFM  ) :

- copy the security strings (Ryndael keys, Cookie keys, etc) from the settings.inc.php of the old shop to the settings.inc.php of the new shop. Without this you won't be able to login. [Note: in Prestashop 1.7 the keys are in parameters.php but at least for the employee password this is not needed]

LE 2 : 

copied \public_html\test16\config\settings.inc.php

to 

\public_html\app\config\parameters.php

cookie values and I have been able to log in backoffice

 

Good to hear.

Let me know when you encounter any problems. I already noticed a lot hooks that should be added.

Edited February 24, 2021 by musicmaster (see edit history)

[dandumit]

13 hours ago, musicmaster said:

Let me know when you encounter any problems. I already noticed a lot hooks that should be added.

Speaking good things. your script migrated product description and attributes correctly ! a paid module didn't !

Well Speaking of issues : 

- something messed between romanian and polish language. probably from install localization menu. now currency name and symbol are messed for eur and ron  

- most annoying until now it's that prices are shown without VAT even if everything looks ok there. Should be a new field or something

When you say : a lot of hooks that should be added. are you reffering to hooks for modules ? 

Thank you,

Daniel 

[musicmaster]

 - That currency language mess-up is interesting as there is a fix in the software for the ps_currency_lang table (it is at the end of copy_shopdata.php). So it would be interesting to know the details. You wont see the signs like € because I don't know where to find them in the PS database. In old versions they are in the ps_currency table. Since 1.7.6 they are in the ps_currency_lang table. But in a period in between they are neither.

- When you add a new product can you assign it VAT? 

- I was wrong about the hooks. That isn't a problem as it isn't among the copied tables.

 

[dandumit]

15 hours ago, musicmaster said:

That currency language mess-up is interesting as there is a fix in the software for the ps_currency_lang table (it is at the end of copy_shopdata.php). So it would be interesting to know the details. You wont see the signs like € because I don't know where to find them in the PS database. In old versions they are in the ps_currency table. Since 1.7.6 they are in the ps_currency_lang table. But in a period in between they are neither.

I had a look on procedure - congratulations very elaborated a lot of things for me to learn, I had a look on tables ps_currency ps_currency_lang.  

Unfortunately on my case those are pretty messed up - I refer to currency description.

some observations in random order : 

- there are many currency in ps_lang than those displayed ( for example gpb )  there are 5 currency displayed in BO and 7 lines in table

- translations of currency are pretty messed up.  I don't know what to do . To correct them manually ? to erase everything and run import again ?

Would it help you if I erase everything and import again ? 

Please see attached picture. It seems that import language package has mapped currency description of Bulgarian and Polish over 2 deleted languages. 

Those translations for currency are imported when importing a language pack ?  I have tried to import again and nothing improved. 

Regarding TAX Rate, this is correct, I had to put as default my country and now it's fine.

[dandumit]

I think that I have found the issue. 

I have truncated everyting,installed presta agin and run import again. 

before script script run in ps_currency_lang were some definitions with some lang id's there starting with id 1 and do not keep count of id's in currency table.

After script run, id's in ps_currency table do not match the ones in currency_lang.

I think that in import localisation there is an issue...

 

Daniel 

[musicmaster]

46 minutes ago, dandumit said:

I think that I have found the issue. 

I have truncated everyting,installed presta agin and run import again. 

before script script run in ps_currency_lang were some definitions with some lang id's there starting with id 1 and do not keep count of id's in currency table.

After script run, id's in ps_currency table do not match the ones in currency_lang.

I think that in import localisation there is an issue...

 

Daniel 

My guess is that you imported country packs and that that created the problematic remainders in the ps_currency_lang table that aren't overwritten. Note that the new shop will use the currency codes of the old shop. So the solution would be that - after you import country/language packages but before you copy from the old shop - you empty the ps_currency_lag table.

I have added a few country configuration entries to the config table (see attachment).

copy_shopdata_config.php

[dandumit]

4 hours ago, musicmaster said:

5 hours ago, dandumit said:

I think that I have found the issue. 

I have truncated everyting,installed presta agin and run import again. 

before script script run in ps_currency_lang were some definitions with some lang id's there starting with id 1 and do not keep count of id's in currency table.

After script run, id's in ps_currency table do not match the ones in currency_lang.

I think that in import localisation there is an issue...

 

Daniel 

Expand  

My guess is that you imported country packs and that that created the problematic remainders in the ps_currency_lang table that aren't overwritten. Note that the new shop will use the currency codes of the old sho

I have tried again to erase everything and run install again. this time I have just added manually currencyes and I have not imported all the language packs. STill mess on currency.

Now i realize that based on language packs that you build at begining of script , existing ps_curency_language has to be updated like :

"5 old languages: bg-bg,en-us,hu-hu,pl-pl,ro-ro (22,1,24,23,14); 5 new languages: bg-bg,en-us,hu-hu,pl-pl,ro-ro (4,1,3,5,2); Language check ok."

update ps_currency_language set id_language= 22 where id_language =4 

............

Thanks

Daniel

[musicmaster]

26 minutes ago, dandumit said:

I have tried again to erase everything and run install again. this time I have just added manually currencyes and I have not imported all the language packs. STill mess on currency.

Now i realize that based on language packs that you build at begining of script , existing ps_curency_language has to be updated like :

"5 old languages: bg-bg,en-us,hu-hu,pl-pl,ro-ro (22,1,24,23,14); 5 new languages: bg-bg,en-us,hu-hu,pl-pl,ro-ro (4,1,3,5,2); Language check ok."

update ps_currency_language set id_language= 22 where id_language =4 

............

Thanks

Daniel

I uploaded a new version of Copy_shopdata. Version 0.43.

You should import the language packs. That is the way to add new languages. Never use the languages page in the backoffice for that purpose! The apcks take care that system words like "next" and "order" are translated.

[dandumit]

On 2/26/2021 at 6:06 PM, musicmaster said:

I uploaded a new version of Copy_shopdata. Version 0.43.

congratulations. your changes did the trick.  shop it's woking now.  

I certify that this script has migrated my shop from presta 1.6.1.8 to 1.7.7.2 .

You have done with this script far better than most acclaimed migration module. 

Small comment on total duration : in script it's presented started date time. but not ad the end. this could be useful for dry run and then migration live schedule/planning.

Small comments on language :

- truncate currency_lang did the trick  - currency are correct 

- but before script / truncate , name of currency was more descriptive like "American dollar" , I still believe that what I have proposed would have work well 

On 2/26/2021 at 5:31 PM, dandumit said:

"5 old languages: bg-bg,en-us,hu-hu,pl-pl,ro-ro (22,1,24,23,14); 5 new languages: bg-bg,en-us,hu-hu,pl-pl,ro-ro (4,1,3,5,2); Language check ok."

update ps_currency_language set id_language= 22 where id_language =4 

Based on your script I will migrade 

[musicmaster]

On 2/26/2021 at 4:31 PM, dandumit said:

"5 old languages: bg-bg,en-us,hu-hu,pl-pl,ro-ro (22,1,24,23,14); 5 new languages: bg-bg,en-us,hu-hu,pl-pl,ro-ro (4,1,3,5,2); Language check ok."

update ps_currency_language set id_language= 22 where id_language =4 

Thank you for your kind words. Please keep me informed if you find more issues.

As for your quoted proposal: this would only work if the same currencies are present in the old shop as in the new one. So either I would need to include a similar requirement as for languages - that the same should be present in the old as in the new I will need to check and repair. But it is an option for future improvement.

Another problem is multiple imports. You would need some system to guarantee that you don't apply the language transformations twice.

 

Edited February 28, 2021 by musicmaster (see edit history)

[musicmaster]

New version: 0.45

Apart from a small fix I have now added a new file: copy_shopdata_images.php. It copies the images from the old shop to the new shop. This works only when they are on the same server.

In addition the recommended approach has been modified:

 - it is now recommended to install the new shop on the server in a subdirectory of the old. 

 - copy this new shop to your pc if you want to do lots of customization and/or experimentation.

In the old approach it was recommended to run copy_shopdata on the pc. But that is quite time consuming. 

[dandumit]

@musicmaster you do amazing things and fantastically those are freee ! 

regarding images copy... I was thinking that at migration would be a good moment to regenerate them and clean old different sizes...

Do prestools suite would help on this ?

[musicmaster]

21 minutes ago, dandumit said:

@musicmaster you do amazing things and fantastically those are freee ! 

regarding images copy... I was thinking that at migration would be a good moment to regenerate them and clean old different sizes...

Do prestools suite would help on this ?

There are several tools in Prestools for that.

With Image-regenerate you can specify that not-used formats are deleted.

Edited April 6, 2021 by musicmaster (see edit history)

[Kornblume]

On 4/6/2021 at 2:09 PM, musicmaster said:

New version: 0.45

...In addition the recommended approach has been modified:

 - it is now recommended to install the new shop on the server in a subdirectory of the old. 

 - copy this new shop to your pc if you want to do lots of customization and/or experimentation.

In the old approach it was recommended to run copy_shopdata on the pc. But that is quite time consuming. 

I know the old documentation and that is very much detailed so even I could follow. Could you please update it to this new method Musicmaster?

For a quick clarification since I have troubles to follow:

Can I also create a new folder for the vanilla fresh installation on 'parallel' to my liveshop concerning the root folder of my host? You suggest subfolder but I like to distinct the liveshop and testshop. You also suggest to create the new one first and then to download/clone it to local PC if many editing is necessary. So I wonder where the benefit over the old approach is.

For my part I intendt to start with 1.6.x fresh and to set up a new template but ofcourse to migrate all business and perhaps necessary module config infos like thos for AdvancedEU module and Paypal etc.

[musicmaster]

On 4/29/2021 at 10:03 AM, Kornblume said:

I know the old documentation and that is very much detailed so even I could follow. Could you please update it to this new method Musicmaster?

For a quick clarification since I have troubles to follow:

Can I also create a new folder for the vanilla fresh installation on 'parallel' to my liveshop concerning the root folder of my host? You suggest subfolder but I like to distinct the liveshop and testshop. You also suggest to create the new one first and then to download/clone it to local PC if many editing is necessary. So I wonder where the benefit over the old approach is.

For my part I intendt to start with 1.6.x fresh and to set up a new template but ofcourse to migrate all business and perhaps necessary module config infos like thos for AdvancedEU module and Paypal etc.

Hi Kornblume,

The old method still works.

The advantage of the new method is that you can work faster. This is specially important when the old shop is no longer usable and you want something running again soon. It requires some ability with Filezilla (renaming and drag-and-drop) that could be better explained yet.

In case you want to take your time to finetune a new theme and new modules this is less important and the old method might be just as well.

If you have a copy of the old manual version that you like can you attach it here to a forum post?

Regards,

M

[musicmaster]

Version 0.49 has two main improvements:

 - copy_shopdata_images is improved and has got some options. The manual has also got some tips on this subject.

 - the carrier tables are no longer copied by default. It didn't work very well when people switched to other carrier modules. Of course you can include them yourself.

[dandumit]

Good Morning,

I am trying now to use "for real" your amazing script to migrate from ps 1.6 to 1.7.6. 

I didn't respect all details from manual and source it's a single shop but destination it's on multishop. 

I am getting error :

MySQL error 1062: Duplicate entry '' for key 'full_shop_url'
Generated by URL '/pts/copy_shopdata.php'
with Query 'UPDATE `ps_shop_url` SET domain='', domain_ssl='',physical_uri='''

I have commented  :

if($table == "shop_url")
  { 
    //$res = dbxquery($conn, "UPDATE `"._DB_PREFIX_."shop_url` SET domain='".mysqli_real_escape_string($conn,$newdomain)."', domain_ssl='".mysqli_real_escape_string($conn,$newdomain)."',physical_uri='".mysqli_real_escape_string($conn,$newdir)."'");
  }

I have commented that line and trying to update manually.

 

On function 

function copy_specific_price_priority_direct($table, $args)
{ global $conn, $oldconn;
  $query = "TRUNCATE TABLE "._DB_PREFIX_.$table;
  $res = dbxquery($conn, $query); 
  if(!$res) 
      sql_error($conn, $query);
  $query = "INSERT INTO `"._DB_NAME_."`.`"._DB_PREFIX_.$table."` (SELECT ".$args." FROM `"._OLD_NAME_."`.`"._OLD_PREFIX_.$table."`)";
  $query .= " ON DUPLICATE KEY UPDATE `"._DB_NAME_."`.`"._DB_PREFIX_.$table."`.id_product=`"._DB_NAME_."`.`"._DB_PREFIX_.$table."`.id_product";
  $res = dbxquery($oldconn, $query); 
  if(!$res) 
      sql_error($oldconn, $query); 
}

 

you are using to execute insert $OLDCONN - I have changed it $conn simple.

 

Thank you,

Daniel 

[musicmaster]

6 hours ago, dandumit said:

Good Morning,

I am trying now to use "for real" your amazing script to migrate from ps 1.6 to 1.7.6. 

I didn't respect all details from manual and source it's a single shop but destination it's on multishop. 

I am getting error :

MySQL error 1062: Duplicate entry '' for key 'full_shop_url'
Generated by URL '/pts/copy_shopdata.php'
with Query 'UPDATE `ps_shop_url` SET domain='', domain_ssl='',physical_uri='''

I have commented  :

if($table == "shop_url")
  { 
    //$res = dbxquery($conn, "UPDATE `"._DB_PREFIX_."shop_url` SET domain='".mysqli_real_escape_string($conn,$newdomain)."', domain_ssl='".mysqli_real_escape_string($conn,$newdomain)."',physical_uri='".mysqli_real_escape_string($conn,$newdir)."'");
  }

I have commented that line and trying to update manually.

 

On function 

function copy_specific_price_priority_direct($table, $args)
{ global $conn, $oldconn;
  $query = "TRUNCATE TABLE "._DB_PREFIX_.$table;
  $res = dbxquery($conn, $query); 
  if(!$res) 
      sql_error($conn, $query);
  $query = "INSERT INTO `"._DB_NAME_."`.`"._DB_PREFIX_.$table."` (SELECT ".$args." FROM `"._OLD_NAME_."`.`"._OLD_PREFIX_.$table."`)";
  $query .= " ON DUPLICATE KEY UPDATE `"._DB_NAME_."`.`"._DB_PREFIX_.$table."`.id_product=`"._DB_NAME_."`.`"._DB_PREFIX_.$table."`.id_product";
  $res = dbxquery($oldconn, $query); 
  if(!$res) 
      sql_error($oldconn, $query); 
}

 

you are using to execute insert $OLDCONN - I have changed it $conn simple.

 

Thank you,

Daniel 

Hi Daniel,

When you make changes it becomes impossible for me to support the script as I don't know the extent of your changes and I didn't test them.

You mentioned

6 hours ago, dandumit said:

MySQL error 1062: Duplicate entry '' for key 'full_shop_url'
Generated by URL '/pts/copy_shopdata.php'
with Query 'UPDATE `ps_shop_url` SET domain='', domain_ssl='',physical_uri='''

What I see here is that all the fields are filled with an empty value. That is not good. But without context I have no idea why that happens.

[dandumit]

good morning. above points are just fine.  I have found an issue : 

values in configuration table for constants : PS_CUSTOMER_GROUP PS_GUEST_GROUP PS_UNIDENTIFIED_GROUP  are different on old shop and new shop. because of that crashes when trying to edit category. what else could be wrong on those configs ?

I see that those values where mentioned above also by someone above on table: ps_configuration:

 

Edited September 13, 2021 by dandumit (see edit history)

[musicmaster]

Hi, you are right. Somehow they had fallen between the tracks in the assumption that they are the same for all shops - what obviously isn't so. It looks like very old shops (PS 1.3?) have different values.

I posted an update.

[dandumit]

Indeed I have started on 1.3 or even earlier.  and struggling to upgrade. I have bought also most expansive module for upgrade but in terms of performance you are far better.

Please have a review of function 

function copy_specific_price_priority_direct($table, $args)   you are doing truncate on newconn and then insert over newconn. is it on purpose like this ?

 

Daniel

[musicmaster]

Hi Daniel,

If you look above you see that the "direct" functions are called when both shops are on the same server. So $oldconn and $conn are the same. The only reason to name one or the other is to stress which table is most important. What really matters in this context is the database and prefix names (_OLD_NAME_._OLD_PREFIX_).

I made a small upgrade in the manual for 0.51.

 

copy_shopdata_on_server.pdf

Edited September 13, 2021 by musicmaster (see edit history)

[dandumit]

Are we there yet ? 

As a Sign of respect for your work and support I have bought your module Prestools Suite. I don't know what they are doing but if those are the same of copy_shop data must be a huge deal !

 

Well your script it's just close to be perfect !  I have a little snag. 

I have used to copy my shop , works flawlessly however a small problem : "no carrier for this order"  i have created manually 2 carriers and those are working. 

Could you please suggest me what to investigate ?  are shop id's messed a bit ? Should I redefine them  ? 

 

Thank you,

Daniel

Edited September 25, 2021 by dandumit (see edit history)

[musicmaster]

3 hours ago, dandumit said:

Are we there yet ? 

As a Sign of respect for your work and support I have bought your module Prestools Suite. I don't know what they are doing but if those are the same of copy_shop data must be a huge deal !

Well your script it's just close to be perfect !  I have a little snag. 

I have used to copy my shop , works flawlessly however a small problem : "no carrier for this order"  i have created manually 2 carriers and those are working. 

Could you please suggest me what to investigate ?  are shop id's messed a bit ? Should I redefine them  ? 

Thank you,

Daniel

Hello Daniel,

Thank you for your purchase.

Carriers are a bit tricky territory. As the system creates a new carrier every time you change the shipping price. I used to copy the carrier settings every time the product and order data were copied. But then I encountered a situation where someone had installed a new carrier module and the next copy erased all its settings. So at the moment the carriers are not copied but you can easily include them in the config file. I probably will switch to only copying the first time.

I don't know the exact situation where you get this message. Are those completed orders? Or do they still need to be shipped? My suggestion would be to include those carriers once.

[dandumit]

Good Morning. Please guide me a little more. 

I have migrated shop , updated few things and tried to complete one order.  Then I hit the issue of "No carrier available". I have tried to edit existing one because I was knowing the behavior of new currier creation. I have tried to play with zones and thing but no success. 

Please tell me what table to include in migration and where. 

Thank you !

Daniel

[musicmaster]

I am a bit puzzled by your question.

Near the top of the config file you will see:

$skip_carrier_tables = true;

Further down you find:

$carrier_tables = array("carrier","carrier_group","carrier_lang","carrier_shop","carrier_tax_rules_group_shop","carrier_zone","delivery","range_price","range_weight");  /* also: module_carrier */
if(!$skip_carrier_tables)
  $copied_tables = array_merge($copied_tables, $carrier_tables);

But you should really be more precise in describing the issue. I assume this is for new orders. Carrier not found is a complicated beast and there even is some bug that can cause it.

Prestools has a tool near the bottom of its Shop Rescue page to check for the most common causes.

[musicmaster]

New in the toolset: Copy_shopdata_fix.

It is meant for shops that used to work but had a bad upgrade.

You install a fresh shop of the same version on the same server - most likely in a subdirectory. You copy the data from the old shop to the new shop like you did with Copy_shopdata. But in this case all tables are copied. 

And when you are ready you assign the new database to the old shop. 

In the process missing fields, changed field formats, missing hooks and missing config values will be fixed.

And it can be done within an hour...

[dandumit]

interesting. I had some issues with some theme and I have did upgrade to 1.7.8. 

then I got : Prestashop 1.7.8 is not yet supported by Prestools.

When it will be upgraded ?  What version do you recommend to run ?

Thank you,

Daniel

[musicmaster]

30 minutes ago, dandumit said:

interesting. I had some issues with some theme and I have did upgrade to 1.7.8. 

then I got : Prestashop 1.7.8 is not yet supported by Prestools.

When it will be upgraded ?  What version do you recommend to run ?

Thank you,

Daniel

I have updated Prestools now to support 1.7.8.

[nikkickc]

Hi

As I can no longer upgrade PrestaShop due to [Ajax / Server Error for action upgradeDb] textStatus: "error " errorThrown:" " jqXHR: " Gateway Timeout"]

 

Just tried this tool out and I get this error -

MySQL error 1045: Access denied for user 'modelho3_pres963'@'localhost' (using password: YES)

My host is telling me that

"I was able to replicate the issue and it seems that the database user does not have permission to write the query's output to a file. The query (a similar one for visual clarity) runs fine when the 'INTO OUTFILE' clause is removed:

MySQL [modelho3_pres963]> select count(*) `id_product_1`,`id_product_2` from `psxz_accessory`; +--------------+--------------+ | id_product_1 | id_product_2 | +--------------+--------------+ | 2443 | 3 | +--------------+--------------+ 1 row in set (0.001 sec)

The exact issue is that the database user is lacking the FILE permission and unfortunately, it can't be granted."

Any help would be appreciated as I have been trying to fix my store (it works but things are wrong in the back office and I can no longer access certain parts!  Have been trying to sort out in fits and starts for the past 18 months! 

Thanks

Nikki