4

I have a server running dovecot using the maildir format with a fairly large userbase (10k users or so) with each user having around a GB quota.

The actual mail directories are stored on a separate storage system that our email server mounts via NFS.

Due to performance issues on our storage system, we are upgrading to a new storage system. We have been rsyncing data from the old system to the new system directly, but this process still takes a rather lengthy amount of time (16+ hrs).

Our plan is to run rsyncs back to back, after the second one finishes, immediately take down dovecot/qmail/etc on the mail server, switch mount locations to the new system, and bring email back up. That'd only total a minute or two for downtime with luck. The problem is any E-Mail that came in while the last rsync was running is still not going to be copied over. So, to alleviate this, after the switch we would rsync again without the --delete flag we typically use. There are some issues with this, but the biggest from my perspective is users not having access to email they JUST had right before the switch.

Does anyone have a suggestion on how to do this with having very minimal downtime still and NOT losing out on the data for any amount of time? The old storage system is a NetApp, the new system is just a freebsd box with a large san attached, and the email server is Ubuntu.

It feels to me like there should be a way to overlap the storage systems from the user's perspective until the migration is complete, but I simply don't have the knowledge on how to do such a thing if it's even possible.

Improve this question
2
  • If you don't have a secondary MX record yet now is the time to setup such a relay server. You can use that to queue incoming e-mail during your downtime, so no errors for remote senders trying to mail your users while you can't accept incoming e-mail on the primary.
    – HBruijn
    Commented Nov 7, 2013 at 16:20
  • @HBruijin We actually have a separate anti-virus/spam device that already queues up mail for us when our server(s) might be offline. Thank you very much for the comment though as I'm sure someone else would benefit from it.
    – Mythics
    Commented Nov 7, 2013 at 17:22

3 Answers 3

Reset to default
3

You can do this with something like Aufs or Unionfs.

Both these filesystems are "union" filesystems. You would do something such as

  • Mount the old NAS at /mnt/old
  • Mount the new NAS at /mnt/new
  • Mount the union filesystem at /mnt/nas with /mnt/new on top of /mnt/old.
  • Any access to /mnt/nas/foo/bar will first look for /mnt/new/foo/bar, and if it's not there, will fall back to /mnt/old/foo/bar. If you modify the file, it will copy the original from /mnt/old onto /mnt/new and then modify the /mnt/new version.
  • After the union filesystem is mounted, you can run an rsync from /mnt/old to /mnt/new. This can be done while the system is live. Accessing /mnt/nas will start picking up the files from /mnt/new as the rsync puts them there.
Improve this answer
2
  • Won't this cause crouption if a queue tries to write at the same time rsync does? Or reversion if a queue writes before rsync (or missing data if rsync is only copying newer)?
    – coteyr
    Commented Nov 7, 2013 at 18:05
  • The only possible problem is if the unionfs creates the file at the exact same time rsync does. But from the scenario in the question this wont happen. However if this answer is applied to other situations, it could be a problem yes. What you might be able to do (I havent tried) is touch every single file on /mnt/nas. The write operation should cause it to copy the file. However it will change all the modification times.
    – phemmer
    Commented Nov 7, 2013 at 20:21
0

How about proceeding like this:

  1. 1st Rsync
  2. Make qmail reply 421 Service not available, closing transmission channel in the banner
  3. 2nd Rsync
  4. Restore qmail's original config (or switch to another server, I'm not sure I got that part)

This way, you'd rely on the fact that clients will retry later to deliver the email.

Improve this answer
1
  • We already queue up mail incoming in a different device so that a response should not be necessary. Our major concern is the downtime of the system itself to our customers trying to access their mail. 16 hrs is simply not acceptable and our current plan could allow immediate access, but not to things they may have just had.
    – Mythics
    Commented Nov 7, 2013 at 17:28
0

I don't think you going to be able to avoid a large amount of down time, while changing your storage backend around, but there are options. The best options however would be tailored to your needs and environment.

If you not going to randomly delete mails you must stop writes to the storage system, either by responding 421, 450, or 452. Personally I would choose 450 "User mailbox unavailable."

So, I would Rsync, followed by returning 450, followed by rsync, and finally the storage change over.

Remember email is not supposed to be reliable. Were just supposed to perceive it that way. A failure to accept a message means that the server sending the message to you should try again. Normally this would be every 4 hours for 24 hours, but that's just "normal" and not a rule.

As an aside if you were not using NFS (or if the NFS server is accessible like this) you could try putting the storage in some kind of cluster, then just adding the new storage to the cluster and remove the old storage. This could be accomplished with either some kind of RAID (if on the same machine) or with something like DRBD. Idea being that you just follow the normal server upgrade path of adding a new server to the cluster, give it time to "catch up" then remove the old server.

Improve this answer
2
  • I'm quite fond of the cluster idea, but I'll certainly have to research it. I'm still a bit new to the unix/linux world, so it may take me some time. We already queue up mail incoming in a different device so that a response would not be necessary. Our major concern is the downtime of the system itself to our customers trying to access their mail. 16 hrs is simply not acceptable.
    – Mythics
    Commented Nov 7, 2013 at 17:27
  • It may have to be, if your system wasn't designed for scaling. Normal server stuff of this size, should be in a cluster. There is no way to safely replicate the data (aside from some kind of cluster) without stopping the writing of new data. If you don't stop new incoming data you risk corruption. Research the cluster, it's your best bet, but will still require down time to setup. However, at least this way, you won't have downtime "next time".
    – coteyr
    Commented Nov 7, 2013 at 18:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .