Question Plesk old->new server migration - best strategy

[burnley]

Hi,
Lately we've been looking at discontinuing the existing CentOS 5 Plesk servers and replacing them with CentOS 7. Given our existing environment, where hundreds of customers with hundreds of GB of data are stored on each server, the basic requirements are:
- Minimal outage.
- The OS upgrade should look like an in-place upgrade. Unfortunately, unlike Debian's vastly superior package management which makes dist-upgrades and release upgrades a breeze, RedHat doesn't support in-place upgrade in versions 5 and 6, this is why we need to be creative. The end result should be a fresh, new CentOS 7 Plesk server with all the settings and customer data imported properly from the old server.

I've recently started to use plesk-migrator for reseller and client based migration between the Plesk servers and, while it's doing a great job with small numbers of clients on large amounts of data, it's pretty slow for batches containing hundreds of clients.
The strategy I'm using currently for the migration is largely based on https://docs.plesk.com/en-US/12.5/m...m-plesk/migrating-via-the-command-line.75722/ with adjustments for our local environment. The main steps are:

- Prepare the source and destination server as documented (ssh keys, generate migration list, import plans, DNS TTL etc)
- Run initial bulk transfer. Keywords: "transfer-accounts --skip-services-checks --skip-main-node-disk-space-checks --migration-list-file ${PLESKMIGRATORLIST} --ip-mapping-file ${PLESKMIGRATORIPMAP}"
- Incremental transfer. Keywords: "copy-content --skip-services-checks --migration-list-file ${PLESKMIGRATORLIST} --ip-mapping-file ${PLESKMIGRATORIPMAP}"
After several incremental transfer rounds I can get a good estimate for the outage window. Based on the that, we schedule the outage window and run:
- Final transfer. Keywords: defer email delivery, "copy-content --skip-services-checks --migration-list-file ${PLESKMIGRATORLIST} --ip-mapping-file ${PLESKMIGRATORIPMAP}", adjust DNS, re-enable email delivery.

For one of the servers, hosting 700+ clients with about that many subscriptions, the incremental transfer phase takes in excess of 3.5 hours, which is way beyond what is acceptable for us. In the console & info.log file I see a lot of what seems to be redundant activity. Is it really necessary to run for every "copy-content" all those commands? Example:
[2016-08-05 13:12:01][INFO] Initialize Plesk Migrator
[2016-08-05 13:12:01][INFO] Load configuration
[2016-08-05 13:12:01][INFO] Target Plesk host: 10.168.1.199
[2016-08-05 13:12:01][INFO] Source Plesk 'pfu' host: 192.168.1.181
[2016-08-05 13:12:12][INFO] START: Check if migration list file exists
[2016-08-05 13:12:12][INFO] FINISH: Check if migration list file exists
[2016-08-05 13:12:12][INFO] START: Check connections
>>> 25 seconds spent here.
[2016-08-05 13:12:37][INFO] FINISH: Check connections
[2016-08-05 13:12:37][INFO] Check migration compatibility of source and target Plesk versions
[2016-08-05 13:12:37][INFO] Check that all required components are installed on source Plesk
[2016-08-05 13:12:37][INFO] START: Fetch basic information about resellers, clients and domains data from source servers
[2016-08-05 13:12:37][INFO] Using the existing shallow dump for 'pfu'
[2016-08-05 13:12:37][INFO] FINISH: Fetch basic information about resellers, clients and domains data from source servers
[2016-08-05 13:12:37][INFO] START: Read migration list
[2016-08-05 13:12:37][INFO] FINISH: Read migration list
[2016-08-05 13:12:37][INFO] Read IP mapping file
[2016-08-05 13:12:38][INFO] Read IP-mapping file /usr/local/psa/var/modules/panel-migrator/conf/ipmapping.conf
[2016-08-05 13:12:38][INFO] START: Fetch information from source panel
[2016-08-05 13:12:38][INFO] START: Fetch configuration data from Plesk servers
[2016-08-05 13:12:38][INFO] Using the existing dump '/usr/local/psa/var/modules/panel-migrator/sessions/migration-session/plesk.backup.pfu.raw.tar' for 'pfu'
[2016-08-05 13:12:38][INFO] FINISH: Fetch configuration data from Plesk servers
[2016-08-05 13:12:38][INFO] Fetch information about APS web applications
[2016-08-05 13:12:38][INFO] Merge information about APS web applications into backup
[2016-08-05 13:12:38][INFO] START: Fetch capability info from Plesk servers
[2016-08-05 13:12:38][INFO] Using the existing dump '/usr/local/psa/var/modules/panel-migrator/sessions/migration-session/capability_dump.pfu.xml' for 'pfu'
[2016-08-05 13:12:38][INFO] FINISH: Fetch capability info from Plesk servers
[2016-08-05 13:12:38][INFO] FINISH: Fetch information from source panel
[2016-08-05 13:12:38][INFO] START: Fetch information from target servers
>>> Why fetch the information again and not reuse the information fetched during "transfer-accounts" stage?
[2016-08-05 13:12:53][INFO] Get target panel's resellers for conflict resolution
[2016-08-05 13:12:53][INFO] Get target panel's service templates for conflict resolution
[2016-08-05 13:12:57][INFO] Get target panel's customers for conflict resolution
>>> More time spend here
[2016-08-05 13:14:14][INFO] Get target panel's auxiliary users and roles for conflict resolution
>>> And here
[2016-08-05 13:16:59][INFO] Get target panel's subscriptions and webspaces for conflict resolution
>>> And here!
[2016-08-05 13:22:34][INFO] Get target panel's subscriptions, sites and site aliases for conflict resolution
[2016-08-05 13:29:23][INFO] FINISH: Fetch information from target servers
[2016-08-05 13:29:23][INFO] START: Convert clients and subscriptions
[2016-08-05 13:29:41][INFO] Convert resellers
[2016-08-05 13:29:54][INFO] FINISH: Convert clients and subscriptions
[2016-08-05 13:29:54][INFO] Convert hosting settings to target panel format
>>> And here!
[2016-08-05 13:49:57][INFO] START: Check target panel capability
>>> And here!
[2016-08-05 13:54:18][INFO] Set component 'mod_perl 'can not be installed in capability report
[2016-08-05 13:54:18][INFO] Set component 'mod_python 'can not be installed in capability report
[2016-08-05 13:54:18][INFO] FINISH: Check target panel capability
[2016-08-05 13:54:18][INFO] Import backup dumps to target panel's repository

So, looking at the above: copy-content started at 13:12:01, but 42 minutes later the migrator still hasn't started to transfer the actual data increments!

Are there any options to speed up copy-content? In the current form we're unlikely to be able to do a full server migration without looking at ~4 hours downtime.
Will follow up shortly with the subscription transfer log data.

 

[burnley]

Example of a domain transfer via copy-content:
[2016-08-05 13:58:01][INFO] [domain.com.au] Copy web files from Unix servers
[2016-08-05 13:58:06][INFO] [domain.com.au] Fix permissions for vhost directory
[2016-08-05 13:58:08][INFO] [domain.com.au] Fix vhost structure for Unix servers
[2016-08-05 13:58:26][INFO] [domain.com.au] Set permissions for statistics and logs directories
[2016-08-05 13:58:26][INFO] [domain.com.au] Fix permissions for web files
[2016-08-05 13:58:29][INFO] [domain.com.au] Adjust common migrated applications for new environment
[2016-08-05 13:58:43][INFO] [domain.com.au] Copy databases
[2016-08-05 13:58:50][INFO] [domain.com.au] Copy mail messages

Particularly slow in my tests are:
"Fix vhost structure for Unix servers"
"Adjust common migrated applications for new environment"
I've seen plesk-migrator spending more than 8 minutes on a wp site with a very large cache in "Adjust common migrated applications for new environment", even though "Copy web files from Unix servers" only took just over two minutes.
What can be done to speed up the server migration?

 

[burnley]

Looking in debug.log for that wp site, here's what I get:
+|2016-08-09_07:12:33,620|I|ST1|core.workflow.runner.by_subscription|domain.com.au||Adjust common migrated applications for new environment
+|2016-08-09_07:12:33,631|D|ST1|plesk.actions.adjust_applications|domain.com.au||Start adjust database connection settings in configuration files of subscription
+|2016-08-09_07:13:24,714|D|ST1|plesk.actions.adjust_applications|domain.com.au||End adjust database connection settings in configuration files of subscription
+|2016-08-09_07:13:24,716|D|ST1|plesk.actions.adjust_applications|domain.com.au||Start adjust paths in configuration and code files of subscription
+|2016-08-09_07:13:24,735|D|ST1|core.application_adjuster.adjuster|domain.com.au||Application path mapping:
=|2016-08-09_07:13:24,735|D|ST1|core.application_adjuster.adjuster|domain.com.au||
+|2016-08-09_07:13:27,239|D|ST1|core.application_adjuster.adjuster|domain.com.au||Exception:
=|2016-08-09_07:13:27,239|D|ST1|core.application_adjuster.adjuster|domain.com.au||Traceback (most recent call last):
=|2016-08-09_07:13:27,239|D|ST1|core.application_adjuster.adjuster|domain.com.au|| File "/usr/local/psa/admin/plib/modules/panel-migrator/backend/lib/python/parallels/core/application_adjuster/adjuster.py", line 153, in _adjust_paths
=|2016-08-09_07:13:27,239|D|ST1|core.application_adjuster.adjuster|domain.com.au|| contents = runner.get_file_contents(filename)
=|2016-08-09_07:13:27,239|D|ST1|core.application_adjuster.adjuster|domain.com.au|| File "/usr/local/psa/admin/plib/modules/panel-migrator/backend/lib/python/parallels/core/runners/unix/local.py", line 66, in get_file_contents
=|2016-08-09_07:13:27,239|D|ST1|core.application_adjuster.adjuster|domain.com.au|| return fp.read().decode('utf-8')
=|2016-08-09_07:13:27,239|D|ST1|core.application_adjuster.adjuster|domain.com.au|| File "/opt/plesk/python/2.7/lib64/python2.7/encodings/utf_8.py", line 16, in decode
=|2016-08-09_07:13:27,239|D|ST1|core.application_adjuster.adjuster|domain.com.au|| return codecs.utf_8_decode(input, errors, True)
=|2016-08-09_07:13:27,239|D|ST1|core.application_adjuster.adjuster|domain.com.au||UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position 139: invalid start byte
+|2016-08-09_07:13:27,240|D|ST1|core.application_adjuster.adjuster|domain.com.au||File /var/www/vhosts/domain.com.au/choosemy/wp-includes/js/tinymce/plugins/spellchecker/rpc.php was skipped when adjusting application configs: it looks like a binary file, or has not-UTF-8 encoding

"adjust database connection settings in configuration files of subscription" takes 51 seconds.
"adjust paths in configuration and code files of subscription" takes a lot more and the code crashes as well. Perhaps too intrusive?

 

[Aleksey Filatev]

Hello!

That's about adjusting of migrated applications for new environment, we have a two things to do: changing of IP-addresses in configuration files of most popular web-applications (adjust database connection settings) and changing of absolute file paths in text files (like .php, .ini, etc.) on migrated subscriptions (adjust paths in configuration and code files). First step is quite fast, because usually there are not much applications installed on migrated subscriptions, and for each application Plesk Migrator usually modify only one configuration file (replace old IP-address with new one), but second step could be much longer, because it depends on count of text files withing subscription.

By default, adjusting of application (both steps) included into copy content task because new files can contain things to fix and we don't know which files was particularly changed after latest synchronization. But adjusting of of applications could be turned off in migration configuration file by adding following lines into [GLOBAL] section:

[GLOBAL]
...
adjust-applications: false
...

After final transfer you can run adjustment once using separate command:

/usr/local/psa/admin/sbin/modules/panel-migrator/plesk-migrator adjust-applications

But do not forget to change migration configuration file and turn on adjustment before run it.

That's about redundant activity at copy content command, unfortunately, you can not skip fetching of data about existing objects from target. Initially it was done to operative check consistency between two servers, source and target, before performing any migration activity to avoid data corruption, but I see that it is actually useless for copy content command, because there no new objects could be created during this operation. I just create a bug PMT-2989 to fix it in next Plesk Migrator release (approximately at the middle of august).

 

[burnley]

Hi Aleksey, thanks for your input. Indeed, the adjust-applications step should only be necessary once, during the initial accounts transfer. Any subsequent incremental transfer should run on the assumption that no new objects have been created in either source, or destination. Also, the adjust-applications step can be sped up significantly if the following assumptions are made:
- The local server configuration doesn't change between the initial accounts transfer and all the subsequent incremental transfers. Included, but not limited to IPs, paths, Unix user & group names.
- Also, between the initial accounts transfer and all the following copy-content operations, no database connection changes are made. Usually this means the db host, db name, username
and password all remain unchanged.
- And the last, but not least: the migration is Plesk->Plesk running the same OS flavour.
If we take into account the above, adjust-applications can / should be split in:
- Adjust vhosts configuration, most of the time should only be the IP addresses and the SSL certificate paths. Only necessary during the accounts-transfer stage.
- Adjust the applications configuration. If the above premises are met, I don't think this step is necessary at all.
To summarize, with the aim of reducing the migration downtime to a minimum, the only mandatory actions copy-content should take are:
- rsync [flags] src dst. For both /var/www/vhosts/ and /var/qmail/mailnames initially. Other paths like /var/lib/psa/dumps can be safely skipped, the admin will sync them later.
- Adjust Unix file ownership. May skip the permissions, assuming they're properly set on the source server and rsync uses '-a' option.
- Dump/restore the databases.
Once the last incremental transfer is complete, I need to adjust the local configuration on the destination server (IPs hostname) on both OS and applications level, then go live.
Thus, the migration downtime will be a sum of:
- Last incremental transfer duration.
- Local configuration changes.
Looking forward to what your thoughts are.