Issue Plesk migrator seems to be duplicating emails

[QWeb Ric]

We have an old server running CentOS 6.7 and a new server running CentOS 7.7. Both servers run Plesk Obsidian 18.0.21 Update #5 but the old server runs Courier mail, and the new is Dovecot.

We have around 80 sites to import so we're working through them one at a time. Scheduling days with our clients to take their sites offline, migrate, and repoint the domains. Most include email addresses and most clients use POP3 for this.

We've imported the first though and after the domain repoint, all of the emails have been re-delivered to the client devices!

I found this Plesk article and it mostly makes sense: POP3 client emails are duplicated after migration - I can understand how the migration from Courier to Dovecot might cause IDs to change and old emails to appear as if new, but I'm not certain about the resolution offered?

Firstly this article seems to be old. It was updated 2 months ago but has comments going back to 2018. It references "a bug with ID PMT-3478 which is already fixed" so if the article is a few years old now, perhaps this isn't the same problem we're experiencing?

Secondly, the article says "run the following on target server, but only after all migration/re-sync tasks are finished" and I'm not sure whether this means migrate a website, run this command, and then repoint the domain, or if it means migrate all websites, click the "finish migration" button, then run this command, and then repoint all domains?

For the record this is the first time I've used the Plesk migrator and at this point I've absolutely no idea what that finish button does. I'm also not sure what the "reload source data" button does and am slightly concerned that it's been a few days since I clicked to start a migration and now the data being migrated might be the data as it was a few days ago rather than the data as it is at the time an individual website is transferred...

Perhaps I'm supposed to start a new migration, pick the one website, transfer, finish the migration, run the maildir-migrate-dovecot command, repoint the domain, and then start another migration for the next website?

The other thing I notice is that the maildir-migrate-dovecot command doesn't reference an individual mailbox or Plesk subscription. It looks to be a server-wide conversion. Is there any harm then, in running this more than once? Again we're trying to import these sites one at a time...

Much appreciated.

 

[QWeb Ric]

There's been no response to this thread yet so I went ahead and gave it a shot. Migrated one of our own websites to test on - SSH'd in, ran the command, and:

/var/qmail/mailnames/[OBFUSCATED_WEBSITE_NAME]/kate/Maildir/courierpop3dsizelist: Broken header: /3 416 1540316444

/var/qmail/mailnames/[OBFUSCATED_WEBSITE_NAME]/info/Maildir/courierpop3dsizelist: Broken header: /3 12 1538215030

/var/qmail/mailnames/[OBFUSCATED_WEBSITE_NAME]/help/Maildir/courierpop3dsizelist: Broken header: /3 766 1385718174


Total: 3 mailboxes / 3 users
0 errors
0 dovecot-uidlist files written

WARNING: Badly done migration will cause your IMAP and/or POP3 clients to re-download all mails. Read Migration - Dovecot Wiki carefully.

Does this mean it's actually a bug in current Plesk builds causing broken header files rather than this being something the migrator script will resolve?

 

[QWeb Ric]

Looking at the code behind the maildir-migrate-dovecot script, this is the part that generates the error:

 if ( $pop3_hdr =~ /^\/2 (\d+) (\d+)$/ ) {
        # /2 <next uid> <uidvalidity>
        $_ = <$f>;
    }
    elsif ( $pop3_hdr =~ /^\/1 (\d+)$/ ) {
        # /1 <next uid>
        $_ = <$f>;
    }
    elsif ( $pop3_hdr !~ /^\// ) {
        # version 0: no UIDs
        $_ = $pop3_hdr;
    }
    else {
        print STDERR "$pop3_fname: Broken header: $pop3_hdr\n";
        close $f;
        return $filename_map;
    }

Ref those regex strings, I'm guessing the problem is that there are "@attachment" folders inside this qmail tree and those regex strings are never going to match an @ character.

What I'm supposed to do about that though, I've no idea =(.

 

[QWeb Ric]

Actually, looking back at those error outputs, I think I was barking up the wrong tree with the @attachment folders.

Furthermore, I'm getting somewhere...

Looking at the migration log within Plesk, I found that "Perform conversion of mail directory files from Courier to Dovecot format" does show up as a task timestamped 20:50 and an ls of the file tree does show that the maildirsize folder was created at that time, but the various dovecot-* files were created at 21:32 which would have been about the time I repointed the domain after giving up with the above migration command.

So, it looks like Plesk tried to run this command to create the dovecot files as part of the migration process but probably invoked the same error I received when running manually. Then I repointed the domain, mail software did a log in, and Dovecot probably created these folders at that time thus sending the emails as if new - because the migration script that was meant to previously convert the files from Courier failed to do so.

At this point I'm convinced that if we can hack this migration script to create the files as it's supposed to, then import another site, Plesk will invoke and this time create the dovecot-* files and when we repoint the domain, all should just go smoothly.

So, looking again at the above regex strings and comparing with the courierpop3dsizelist file, I've spotted that the first line of this file is prefixed with a "/3" and that those regex strings are looking for a prefix of "/2" or "/1" and falling back onto a "version 0" when it's neither. This appears to be the issue - our courierpop3dsizelist files referencing the "/3" must be a "version 3" that this script isn't built for.

The full line in this case, is "/3 766 1385718174" which I'm guessing matches the version 2 line - "<next uid> <uidvalidity>"... So "version 3" might actually be the same format as "version 2" and I can just hack this code accordingly?

Slightly conscious that this might not be a safe edit and that this is a production server, but I feel closer to a resolution now.

PLEASE, is anybody reading this who can offer some input?

 

[Brujo]

well since you can repeat the issue you can fill a report. I suppose you use the latest available version of the migratior extension - 2.18.0 (20 January 2020) .

 

[QWeb Ric]

Hi Brujo,

Thanks for the link, I'll file a report shortly.

It's 2.18.0.897, yes. And Plesk is 18.0.21 Update #5 on both servers.

I did dig further in to my above findings actually, and discovered that /3 seems to coincide with the introduction of Unicode / UTF8 support in Courier 5 so presumably the UID format had to be changed to be UTF8 compatible. So it looks like the migration script Plesk uses isn't compatible with UTF8 enabled Courier versions.

I also found that dsync is the preferred method these days. It's what Dovecot advise to use, but I've not had chance to dig any further into that and I couldn't find a Plesk specific article on migrating mail using dsync.