Resolved Migration doesn't properly migrate courier to dovecot. Subscriptions are lost!!

[mr-wolf]

After a migration I always need to check the webmail.
I'm getting the Plesk standard page on the target server.

Even though "roundcube" is selected on the target server I need to switch it to "Horde" and then back to "Roundcube" to get it working again.


Then I also need to enable the subscription for all the extra IMAP folders as only the standard ones are subscribed to.
Just migrated 1 domain for which I needed to subscribe to > 800 folders.

Edit:
I have many more domains to migrate and will check each migration. I think I will soon know if it does this always or on occasion. I have noticed this more in the past and that was with the old migrator extension, so it may be an old bug.


Another thing I noticed is that this domain has lost its original "setup date".
The domain I migrated this morning has a setup date of today.
I'm used to it that Plesk keeps the original setup date. I have some domains as old as 2007 although the oldest Plesk server is only 4 years old.
I will open another thread for this

 

[mr-wolf]

I think you did a great job on the migrator.
The way it works now is a great improvement on how it used to work.
Keeping the migration process open and allow for a quick resync is exactly what we need as ISP's to try and have a transparent migration for the client.

It would be nice though if you went for this extra mile and fix this anomaly so I don't need to fix


Before migration:

Roundcube is working:




After Migration:

In the example I'm preparing I didn't need to turn off / turn on webmail to get it working on the target server.
It sometimes works as it should, but also sometimes doesn't.
This time that part worked as it should have (I couldn't know this beforehand).
As we have a lot of customers that don't use webmail this often goes unnoticed.

What is a bigger problem is this.
All the non-special folders have their subscription turned off in IMAP.
The client doesn't see them anymore and thinks they are lost.
One client phoned me and saw them disappear in front of his eyes.

I can turn on the subscription again in the webmail and then it will also work for his email-client.

You may want to know that the target server works with Dovecot and the source server with Courier.





On this account (different address of same domain) I never tried the webmail.
Thunderbird never connected to this domain before and upon its first connection it shows none of the folders as subscribed.
I can only assume that on my client's account that only the non-special folders disappear.




When I connect with Roundcube it seems it subscribed to all its special folders.

 

[mr-wolf]

(No more than 8 pictures are allowed per post)



When I then connect again with Thunderbird it has subscribed to these Roundcube special folders





After turning on all the subscriptions in Thunderbird they will also be subscibed in webmail.


Even if this gets fixed there's another anomaly that would make these migrations less than perfect.
A lot of users are using Outlook and these MUA's create special folders like "Verwijderde items" (for trash) and "Verzonden items" (for Sent).
To make less of a mess I always change the special folders in Roundcube to those folders as Outlook can't change them.
Then I delete the folder "Sent"
I can't remove the folder "Trash", so I merely unsubscribe.

All this work will also gets reverted by this migration.

It would be nice if you confirmed reading this (Never got a reply on my first post) and if you try to repeat such a migration and check if it happens to you, too.

 

[Bitpalast]

Regarding the Thunderbird issue: I think this is caused by the changing user ID on migration and is questionable whether that behavior can be changed by Plesk. When the uid validity of a folder changes, Thunderbird sets the folder to "offline". This is not really caused by Plesk or the target mail server, but by Thunderbird. On migration, a new user account is created on a new server, and owernships of files and folders are set to this new account accordingly. The user ID of the new account can only accidentally be the same as on the old server. Normally it will be different. Now, as the folders do no longer belong to the uid that Thunderbird had stored, it does not automatically subscribe to them.

 

[mr-wolf]

Hi Peter,

Thanks for answering.
However....

All Subscriptions are dictated by the server, as is proven by the examples in which I can turn on subscriptions for Thunderbird to on by doing this in Roundcube as I can do it the other way around. If I turn on a subscription in Thunderbird, they start to appear in Roundcube.

The example I gave with Thunderbird (no subscriptions turned on) was a maiden install of that account after the migration.
It completely filled the INBOX, which is normal (although it showed unsubscribed).
But NOT ONE of the other folders got synced (I then deliberately didn't subscribe to them in Thunderbird).
I stopped/started Thunderbird several times. Gave a "refresh" in folder, but none of the folders would "turn on".

I then stopped Thunderbird, logged into Roundcube and saw it subscribed to all the "special folders"
I then started Thunderbird and suddenly those same folders were subscribed, but still not the Roundcube-non-special-folders.
I closed Thunderbird again and in Roundcube I turned on the subscription to all folders.
After starting Thunderbird it was subscribed to all folders.

This disproves your hypothesys that the server doesn't control the subscriptions that are used by Thunderbird.

When I create a new account on a Plesk server and login to webmail all these folders are created and subscribed to. If I install that account in Thunderbird it will show all these folders subscribed to on initial start.

I'm using Plesk as of 2004 and in all this time I never noticed that subscriptions were turned off after migration.
This all happened since using the new migration tool ( which has improved in many other aspects I would like to add again).

This new migration tool also has been updated quite recently, so maybe this wasn't happening before and you may not have migrated anything in the last week.

 

[mr-wolf]

I would like to summarize the migration issues like this:

1. webmail.<domain> shows the Plesk page after the migration. Turning webmail off/on fixes this. This doesn't happen always.
2. IMAP-subscriptions are turned off after the migration. Some mailclients (and also the webmail) have mandatory folders, so they get their subscription turned on. Typically the other folders will never get subscribed without manual intervention.

 

[mr-wolf]

Migration is giving me much more work this way.
I need to login quickly onto webmail (sometimes needing to disable/enable roundcube) after changing DNS and start subscribing to all the folders.

If I don't do this it means that my clients will see all their folders disappear in their MUA's.......

Please fix this!!!!

 

[mr-wolf]

I wrote this quick-and-dirty script just now to check which subscriptions are not applied in Dovecot.
I also tested with a config parameter "autosubscribe", but that didn't do the trick.

Maybe I can improve on this script by writing that subscription file instead of only showing stuff.
First I need to examine more.

Please check the migration tool....
I have a lot of extra work with these 150 domains I'm migrating.
Some people even have more than 60 mailfolders....

As I see how this subscription is arranged it becomes more and more clear to me that this is a "bug" or "oversight" when migrating from courier to dovecot.
These 2 programs use different files to keep their subscription.


# cat /usr/local/sbin/unsubscribed

#!/bin/sh

TMPDIR=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`


MAILDIR=/var/qmail/mailnames/

find ${MAILDIR} -maxdepth 3 -name Maildir >${TMPDIR}/accountroot

while read ACCOUNT ; do
  if [ ! -f ${ACCOUNT}/subscriptions ] ; then
    echo "${ACCOUNT} has no subscriptions"
  else
    echo ""
    find ${ACCOUNT} -type d -mindepth 1 -maxdepth 1 | grep -v courier |  sed 's/^.*\/\.//g' | egrep -v '(tmp|cur|new)$' | sort >${TMPDIR}/folders
    sort ${ACCOUNT}/subscriptions | grep -v "INBOX$" >${TMPDIR}/subscriptions
    diff ${TMPDIR}/folders ${TMPDIR}/subscriptions >${TMPDIR}/difference

    if [ -s ${TMPDIR}/difference ] ; then
      echo ${ACCOUNT}
      cat ${TMPDIR}/difference
    fi

  fi

done <${TMPDIR}/accountroot

rm -r ${TMPDIR}

 

[mr-wolf]

I am very disappointed that no reaction is given in this thread by Plesk.
At least some confirmation that they agree it shouldn't work the way it is doing now.

I succeeded in creating a procedure for migrating Plesk courier to Plesk dovecot which involves a script that enables all the subscriptions of the folders.
This is not the same as mimicking/copying the subscriptions of the source server, but it beats, hands down, the current situation in which all folders are unsubscribed.

For my clients this means that they see their mail disappear in front of their eyes.
I already had a few calls from people in panick.

That this script works is also proof that it is controlled server-side.
Because the server from which I'm migrating turns off in 2 weeks I can't wait for a solution from Plesk.
I still need to do some 45 domains.

It's hours of work. My script takes care of it in 1 second.
Yes, it took some time to write, but I now also know for certain that all folders are subscribed.

For those in the same boat:


# cat /usr/local/sbin/unsubscribed

#!/bin/bash

MODIFY=
CREATE=
MODIFIED=0
CREATED=0
DOMAIN=".*"

while getopts mcd: OPTION
do
  case $OPTION in
    c)   CREATE=true;;
    m)   MODIFY=true;;
    d)   DOMAIN="$OPTARG";;
    ?)   printf "Usage: %s: [-c] [-m] [-d <domain> ]\n" $0
    exit 2;;
  esac
done
shift $(($OPTIND - 1))

if [ -n "$*" ] ; then
  printf "Usage: %s: [-c] [-m] [-d <domain> ]\n" $0
  exit 1
fi

TMPDIR=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
DOVECOTROOT=/var/qmail/mailnames/

find ${DOVECOTROOT} -maxdepth 3 -name Maildir | sort >${TMPDIR}/accountroot

while read ACCOUNT ; do

  if echo "${ACCOUNT}" | grep -q "\/${DOMAIN}\/" ; then

  if cd ${ACCOUNT} ; then

    NOMAILDIR=${ACCOUNT%/*}
    MAILBOX=${NOMAILDIR##*/}

    NOMAILBOX=${NOMAILDIR%/*}
    DOMAINNAME=${NOMAILBOX##*/}

    echo "************** ${DOMAINNAME} -- ${MAILBOX} ****************"

    # Create a subscription file containing all folders
    find . -mindepth 1 -maxdepth 1 -type d | grep -v courier |  sed 's/^.*\/\.//g' | egrep -v '(tmp|cur|new)$' | sort >${TMPDIR}/folders

    FILLED_TRASH="`find ".Trash/cur" -type f 2>/dev/null | head -n1`"   #  Check folder Trash if it exists
    if [ -z "${FILLED_TRASH}" ] ; then                                  #  Is the folder Trash empty??
      if egrep -iq "(deleted|verwijderd|prullenbak)" ${TMPDIR}/folders ; then
        sed -i '/Trash$/d' ${TMPDIR}/folders                           #  if an alternate folder exists for deleted items then unsubscribe Trash
      fi
    fi

    if [ ! -f ${ACCOUNT}/subscriptions ] ; then
      echo "${ACCOUNT} has no subscriptions"
      cat ${TMPDIR}/folders
      if [ ${CREATE} ] ; then
        let CREATED+=1
        echo "Create file subscriptions"
        cat ${TMPDIR}/folders >./subscriptions
      fi
    else
      sort ${ACCOUNT}/subscriptions | grep -v "INBOX$" >${TMPDIR}/subscriptions
      diff ${TMPDIR}/folders ${TMPDIR}/subscriptions >${TMPDIR}/difference

      if [ -s ${TMPDIR}/difference ] ; then
        echo "Diff : \"`cat ${TMPDIR}/difference`\""
        if [ ${MODIFY} ] ; then
          let MODIFIED+=1
          echo "Replace file subscriptions"
          cat ${TMPDIR}/folders >./subscriptions
        fi
      fi
    fi
  fi
  fi
done <${TMPDIR}/accountroot

echo "for ${MODIFIED} mailboxes the file \"subscription\" has been modified"
echo "for ${CREATED} mailboxes the file \"subscription\" has been created"

rm -r ${TMPDIR}

 

[mr-wolf]

By creating a script that solves my problem it is proven that the migration of courier => dovecot can be fixed server-side.
It is now broken!

I am unable to understand why no comments are given to this thread.
Not even from those that have the same use-case but are not seeing this behaviour.

Is no-one at the moment migrating mail from a courier server to a dovecot server?
Courier was always the default mailserver of Plesk and had now been replaced by Dovecot.
This makes me think that many more are doing this kind of migration.

Although it is easily fixed, the impact of the unsubscribed folders is heavy to the user of those mail accounts.
Some have seen their mail disappear in front of their eyes.
As this script proves this is totally unnecessary and it should be given priority to the developers that have written the migration script.

Again I want to stress that I think that the migration software is impressively good and much improved.
I would not want to go back to the old one although I was quite satisfied of that one too.

My script only puts all folders to subscribed. The Plesk migration tool should of course only set the subscriptions to "subscribed" if they are that way on the old server.

Please at least react!!!

 

[Alina Dmi]

Hello, mr-wolf!
Please, fill out this form https://www.plesk.com/professional-services/free-migration/ we`ll investigate it.

 

[mr-wolf]

Hi Alina,

Thanks for reacting.

As things are now I can cope doing the migration myself by using the script I wrote.
I just do a migration of 10 domains.
After the migration I run my script "unsubscribed -c"
This will create all the subscriptions for the newly created domains on the target server.
Then I go through each domain and give it a resync on mail only and change the DNS.
After the TTL (which is set low) I give them another resync (some time later another to be sure).

As the migration script is not aware of the file I wrote, this will not influence the migration itself.

BTW. The link you gave gives me a 404

The 2nd bug is a minor one and I shouldn't have taken it into the same thread. Sometimes the webmail is not enabled after migrating. I need to disable/enable it to get it going. It doesn't do this always, though.
The main bug is totally reproducible.

 

[Alina Dmi]

BTW. The link you gave gives me a 404

Ohh... I`ve informed guys, sorry

 

[Shao Miller]

I could be mistaken, but I believe the following will "fix" a Courier-to-Dovecot "migration"/"conversion"/whatever, specifically with respect to IMAP "subscribed folders":

cd /var/qmail/mailnames/
for i in $(find . -maxdepth 3 -type d -name Maildir); do sed 's/^inbox\.//gI' $i/courierimapsubscribed > $i/subscriptions; done
find . -maxdepth 4 -type f -name subscriptions -exec chown popuser:popuser "{}" \;

 

[Mr.D]

Hello!

This issue was fixed in Plesk Migrator release 2.7.
Now there is no need to perform additional actions after migration from courier to dovecot.

 

[mr-wolf]

I could be mistaken, but I believe the following will "fix" a Courier-to-Dovecot "migration"/"conversion"/whatever, specifically with respect to IMAP "subscribed folders":

cd /var/qmail/mailnames/
for i in $(find . -maxdepth 3 -type d -name Maildir); do sed 's/^inbox\.//gI' $i/courierimapsubscribed > $i/subscriptions; done
find . -maxdepth 4 -type f -name subscriptions -exec chown popuser:popuser "{}" \;

This is six months after the introduction of the bug, my discovery of it, my workaround (had several), my reporting and the subsequent fix by Plesk....

What was the meaning of your post?

 

[Shao Miller]

This is six months after the introduction of the bug, my discovery of it, my workaround (had several), my reporting and the subsequent fix by Plesk....

What was the meaning of your post?

1. I don't see Plesk having addressed this anywhere in the thread until after my post. For that reason, I had no idea that this had been addressed. And I also have no idea why my recent Plesk servers still have the issue. Perhaps they are somehow out of date.
2. My work-around is very specific in that it doesn't just auto-subscribe all folders like some of the other code in this thread, but attempts to carry over just the Courier subscriptions.

If it's been fixed and I simply missed something in this thread about that, then I'll simply apologize for the noise.

 

[mr-wolf]

Hi Shao,

In retrospect I found my post somewhat blunt for which I want to apologize.

If you don't see this fixed on your system you should check the version of the migration extension.
For a while the migration tool is an extension and not part of the Plesk core.
It is possible yours isn't updated.

Let me know....
I'm very interested.

 

[Shao Miller]

My version of "Plesk Migrator" is 2.11.4.

 

[mr-wolf]

Well, that's the one I'm using...
But I have migrated all my Courier-accounts at the beginning of this year, so I'm "All Dovecot".
No way to test this anymore, but I'm 99% sure I've seen it working properly.

Can you do a test and really make sure the migrator is not making the necessary files (/subscriptions)??

I reported this in May of this year: Forwarded to devs - Courier to Dovecot migration does not keep subscribed folders

The report included my suggestion to use this sed during migration:

sed '/^INBOX$/d;s/^INBOX//g' courierimapsubscribed >subscriptions