Forwarded to devs Mail not migrating properly if source server is Courier 5

[QWeb Ric]

User name: QWeb Ric

TITLE

Mail not migrating properly if source server is Courier 5

PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE

Plesk 18.0.21 Update #5 and Migrator 2.18.0-897. CentOS 7 on the destination server, CentOS 6 on the origin server.

PROBLEM DESCRIPTION

When migrating a website from a Plesk server running Courier mail, to a Plesk server running Dovecot mail, emails are migrated successfully but the client software then downloads all existing emails again.

I've narrowed this down to an issue with the maildir-migrate-dovecot script which seems to be based on old Dovecot migration scripts and is presumably what Plesk automatically runs during the migration process. This script appears to be built to convert lines from courierpop3dsizelist into a Dovecot format, but only works if the courierpop3dsizelist file begns with "/2" or "/1" (UID format version numbers, I believe). With Courier 5 this file begins instead with "/3" which looks to have been introduced inline with Courier 5 receiving UTF8 support.

In a nutshell, the migration doesn't properly create the Dovecot UID maps and as such, all migrated mail is considered to be new and re-downloaded.

The format of UID lines within this file may or may not differ to /2 formatted files. I'm not able to investigate this unfortunately.

STEPS TO REPRODUCE

Begin a migration process for a website where the source server is running Courier and its courierpop3dsizelist file begins with /3, and where the destination server is running Dovecot.

After migration, check for the existence of dovecot-* map files alongside the original courierpop3dsizelist file. Such files should, but don't, exist.

Repoint the domain to the new server and check for new mail using a standard mail client. Existing mail will be re-downloaded due to the missing dovecot-* map files.

ACTUAL RESULT

dovecot-* map files don't exist due to the migration script not being able to interpret the courierpop3dsizelist file if version 3 is specified. More precisely, the script outputs a "Broken header" error if run manually, which is produced by the part of the code that tries to interpret this file.

EXPECTED RESULT

dovecot-* files should be produced and existing mail should not be re-downloaded after the domain repoint.

ANY ADDITIONAL INFORMATION

I originally posted to the forums here, where I've posted more detail throughout my investigation: https://talk.plesk.com/threads/plesk-migrator-seems-to-be-duplicating-emails.354914/

Dovecot these days seem to advise using dsync scripts instead of the migration scripts that Plesk seems to incorporate. maildir-migrate-dovecot itself hasn't been updated for a number of years now, by the looks of it.

We're currently paying for more servers than we need as I'd hoped to have all websites migrated by now, but can't migrate any client site that contains mailboxes at the moment due to the risk in them having to receive potentially thousands of old emails when the domains are repointed, so would obviously appreciate any help in resolving this issue please while a proper patch is produced for Plesk. Presumably either the maildir-migrate-dovecot script needs hacking to support /3 versioned courierpop3dsizelist files, or this script needs replacing with a dsync based alternative.

Much appreciated.

YOUR EXPECTATIONS FROM PLESK SERVICE TEAM

Help with sorting out

 

[QWeb Ric]

Is there any update ref this one please? We're currently unable to finish migrating websites from our older servers which are costing a small fortune to keep running =(.

 

[IgorG]

Bugreport PMT-4721was confirmed and submitted. Developers are working on it for including a fix in the one of nearest Plesk Migrator update.

 

[QWeb Ric]

Excellent, thanks Igor. Any idea of a rough ETA for that?

 

[QWeb Ric]

Sorry to nag, but we're still having to run old servers in parallel with the new because we're unable to migrate the sites that have mail accounts associated to them. Could we get an ETA on when this update will be pushed out please?

 

[IgorG]

Could we get an ETA on when this update will be pushed out please?

Sorry, no ETA.

 

[QWeb Ric]

Damn. Not even a loose one? Is it likely to be ready in weeks or are we talking months?... Years?

We're literally paying a couple of hundred GBP per month more than we need to at the moment, for servers hosting websites that we expected to be migrated by now, and this bug is the only thing stopping them from being. Even an unofficial, barely tested patch that I can hack in as a temporary fix would suffice quite frankly.

 

[QWeb Ric]

I see Plesk Migrator 2.19.0 was released a couple of days ago with an intriguing changelog entry:

After migrating mail, actual dates when emails were received are no longer changed to the date of migration on certain mail clients (for example Apple Mail). (PMT-4559)

Is there any chance PMT-4721 self resolves because of this, or was it an entirely different issue?

We're still having to pay to keep these old servers online and would appreciate any updates you can provide on this. Is there not a public bug tracker we can watch?

 

[IgorG]

PMT-4721

The fix is expected in version 2.20.0 of Plesk Migrator.

 

[QWeb Ric]

Brilliant, thank you!

 

[QWeb Ric]

Any eta for Plesk Migrator 2.20 yet? There's been a couple of minor versions since March 16th but no sign of a PMT-4721 fix within them...

 

[QWeb Ric]

This is getting ridiculous now. It's been 4 months since I opened this thread, 2 months since it sounded like we were getting a fix, and almost a month since I asked for an update but still the patch isn't released and there's no eta?

 

[QWeb Ric]

Almost another month has passed. That's 5 months now, paying for servers we no longer need because migrating mail accounts from them isn't working...

 

[QWeb Ric]

I've just set up a malbox on one of the old servers, sent a test email to it, migrated, and it doesn't look like this email has duplicated this time.

Migrator 2.20 isn't released yet and PMT-4721 isn't referenced in the changelogs yet, but PPPM-11776 fixed back in April sounds suspiciously similar to the above. Was this actually all resolved back in April and I've been left waiting with no update all this time?

 

[Dave W]

Hi Ric,
We are seeing this issue as well, have you had any success in resolving?
Rgds
Dave

 

[QWeb Ric]

Hi Ric,
We are seeing this issue as well, have you had any success in resolving?
Rgds
Dave

Hi Dave. Our migrations have been working since my above reply on June 22nd 2020. Just a couple of days ago actually, we finished migrating the last account from our oldest server and this one had a 10gb mail inbox associated to it. No problems at all with the migration.

The servers we're migrating from are CentOS 6 running Plesk Obsidian 18.0.31 Update #3. This was the last update we received to these older servers as CentOS 6 is no longer maintained. The server we're migrating to is CentOS 7 currently running Plesk Obsidian 18.0.35 Update #2. I'm not sure exactly which version fixed the original issue but presumably if you're running Plesk newer than June 2020, you should already be good.

 

[Dave W]

Hi Ric,
Ive a ticket in with the developers. We're in the same boat moving off C6, all working well apart from this issue.
Thanks for the reply.

 

[QWeb Ric]

Hi Ric,
Ive a ticket in with the developers. We're in the same boat moving off C6, all working well apart from this issue.
Thanks for the reply.

I take it the new server is running a recent enough Plesk version?

Looking at our servers, /usr/lib64/plesk-9.0/maildir-migrate-dovecot (which I believe is the script that was at fault for us), was last updated April 27th so presumably if your Plesk was updated after that date, you'll have the latest version.

When I originally examined this file and opened this ticket, I'd discovered that it was built to expect to find /1 or /2 at the top of the courierpop3dsizelist file and this number defines the protocol version used, or something like that. Looking at maildir-migrate-dovecot now, if you scroll down to the read_courier_pop3 function there's a if ( $pop3_hdr =~ /^\/[23] (\d+) (\d+)$/ ) { line which I believe recognises /3 at the top, fixing the issue we had.

Check that file for the same and if the line is present, your issue probably isn't the same as ours was.

Best of luck.

 

[Dave W]

Hi Ric,
Yeah shiny new plesk box, I've narrowed the issue down to a difference in the uidl formats between the source and destination servers. They look similar but with the last 5 digits of the uidl differs between the source and destination server. More digging to be done.
Cheers
Dave