rsync on running home dir causes infinite run

Question

I have a rsync set to run every morning at 5am.

sudo rsync -xrltDu --delete /home/mirthen/ /long/backup/folder/name/

It works for the most part, however it will once in a while hang and never finish when files change on the source system. For example .mozilla which contains my firefox settings and could change files, remove files, add new files all while rsync is running and building its file list for copy.

I will usually get messages about a file vanishing, which is fine as they were temp files in most any case. There has never been any actual error leading me to know why rsync never finishes its copy/backup. It will run, might pop a message about a file vanish, maybe 10+ such messages, and then just sit there taking up a little bit of cpu/mem. Then the next day at 5am i get yet another set of rsync processes running again, and the same exact thing happens.

I have run a full rsync using the same command, and while the system is doing all its file changes, and had it complete just fine. Just a good percent of the time it will hang and not give me a clue as to why.

Is there some way to get it to do the backups on a live system without hanging up, something i am missing. I did have it previously set to do 'rsync -au --delete ....' however moving it across the network to a system with different users I wanted to just skip it trying to keep group/user settings.

There is of course more then just the mozilla folder that changes during the rsync, quite a few other processes running at any given time that will constantly be changing at 5am.

If it matters, Linux Mint KDE 17.2 64bit. Destination is Ubuntu Server 14.04 64bit

Answer 1

The proper way to do backups on a live system is to use a snapshot of the filesystem (I don't mean that the snapshot is the backup, I mean that it can be used to save the files without anything moving). This can be easily achieved if you have BTRFS, LVM, or ZFS, but otherwise isn't practical.

Alternatively, the best approach is to simply ignore the files that change often -- typically they do not contain valuable data. My backup deliberately skips my browser's cache directory. rsync has --exclude options that can achieve this quite easily.

The best way to diagnose what rsync is doing when it hangs is to attach strace:

strace -p `pgrep rsync`

Depending on your OS, you might find that that gives permission denied. On Linux, the usual solution to this is to enable debug permissions:

sudo sh -c "echo 0 > /proc/sys/kernel/yama/ptrace_scope"

I think that setting only applies to new processes that start with the new setting, so you should set it before the rsync task starts. The setting will not persist across reboot. Once you've figured out what's going on you should set ptrace_scope back to 1, as it's a security risk (1 means you can only attach a debugger to child processes, like a debugger normally would).

Answer 2

Check to make sure none of the folders in /long/backup/folder/name are mounted on /home/mirthen. You may be stuck in a recursive loop!