Powered by Blogger.

Moving to Google Apps with imapsync

imapsync seemed to be the tool of choice for a lot of people doing migrations of IMAP users, and it has done a very good job here too. I used version 1.311 from source rather than the older package provided with Ubuntu at the time as it contained various fixes for Google's IMAP servers.

Moving standard folders over to Google Apps using imapsync is pretty straightforward and can be done without any special consideration:

 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 imap.gmail.com --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN --port2 993 --ssl2
Just running the command as above doesn't account for any naming differences between the old imap server and Gmail, this is easily taken care of with the --regextrans2 option which allows regular expressions to be applied to folder names as they are copied.
The old sever kept it's sent mail in a folder called "Sent Items", so a first attempt at renaming was:
 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 imap.gmail.com --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN --port2 993 --ssl2 \
    --regextrans2 's/Inbox\/Sent$/Sent Mail/'
This unfortunately led to a label being created in the Gmail web interface called "Imap/Sent Mail" - not the desired result. The mail should end up in the actual "Sent Mail" folder.

Sent Mail not Imap/Sent Mail 

Having a look at the gmail account in Thunderbird, the standard gmail folders show up under a parent folder called [Google Mail]. Armed with this information a quick google search reveals a few posts about using the --prefix2 option to get the mail to sync into the correct place - it basically boils down to putting the existing sent mail into a folder called "[Google Mail]/Sent Mail". Because of needing to use the --prefix option the sync of the mail was split into two passes.

Two pass email syncing 

  1. Sync over all the folders in your current imap account, but make sure you exclude your sent, draft, junk and deleted items
  2. Sync over the sent mail, drafts and if needed deleted items (you may have somebody who uses their deleted items folder as a filing system - beware!)
So the two pass solution becomes:
 #transfer the standard folders, skipping sent, drafts, deleted and spam
 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 imap.gmail.com --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN --port2 993 --ssl2 \
    --exclude "Sent Items|Deleted Items|Drafts|Spam" \
 #transfer the [Google Mail] folders
 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 imap.gmail.com --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN --port2 993 --ssl2 \
    --folder "Inbox/Sent Items" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\/Sent\ Items$/Sent Mail/' \
    --folder "Inbox/Deleted Items" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\/Deleted\ Items$/Bin/' \
    --folder "Inbox/Drafts" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\/Drafts$/Drafts/' \
 #add the following to the lines above if you want to keep your spam!
     --folder "Inbox/Spam" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\/Spam$/Spam/' \

A localisation gotcha 

One word of caution - the "[Google Mail]" part as well as the actual folder names will change depending on the language set for the account. With the language set to US English (the default) rather than British English the folder prefix was [GMail] and some of the folder names were different e.g. Trash rather than Bin.

Incremental Syncing 

As the sync took the best part of two weeks, it was run multiple times. One by one users were switched over to Google Apps. imapsync really comes into it's own at this point as it can be run multiple times and will not re-transfer email. The --skipsize option is needed if you re-run imapsync otherwise due to the way the Google imap works the messages will not be regarded as the same and will be re-transferred.

The little details 

The final version of the sync script ended up being a bit more complicated as a result of having to deal with the way the Outlook plugin a few people were using worked.
Skipping the Outlook folders was easily done with:
 --exclude "Tasks|RSS Feeds|Outbox|Junk E-mail|Junk|Junk Email" \
 --exclude "Notes|Journal|Calendar|Contacts|Sent Items|Trash|Drafts|Sent$|Deleted Items|Deleted Messages|sent-mail|Sent Messages" \
 --exclude "Sent Items Imap|Drafts Imap|AVG Virus Vault" \
The Kolab plugin was storing the email inside of the mail IMAP INBOX in a folder called Inbox i.e. INBOX/Inbox. This meant that for the users who had been using Outlook the command line to transfer the mail boxes change slightly to account for the extra Inbox folder:
 --folder "Inbox/Sent" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\/Sent$/Sent Mail/' \
Syncing over the main Inbox to the correct place has to be done in a separate pass:
 --include "INBOX/Inbox" --regextrans2 's/Inbox$/INBOX/' \


All in all imapsync got 12 users and 39GB of data migrated in a very painless fashion and allowed people to be moved over to Google Apps one at a time as and when their mail was synced. So far nobody has missed anything!
    Blogger Comment
    Facebook Comment