Rsync rotating logfiles

Question

currently I have an troublesome problem. I needed quite a time for identifying this problem. Anyhow that's what I'm doing:

I collect log files with rsync from multiple servers to archive them. Its important to preserve the time attributes so I'm using the archive-mode of rsync. But in some cases this leads to some errors.

The problem is that the log files are rotating (changing) on the servers.

So current logs that are in use are named like test.log (they are not getting rsynced). After some time and/or a specific file size the log file is rotating. Meaning test.log is renamed to test.log.1. Now when test.log is rotated again, test.log.1 is renamed to test.log.2 and test.log is renamed to test.log.1.

I think when test.log would go directly to test.log.2, there would be no problem. But that's something I cannot change.

And exactly this rotation happens somehow (sometimes) exactly when I'm rsyncing. So rsync get all the time-attributes of the files, then the rotation happens and then rsync starts to collect the logfiles. But now for example test.log.1 isn't the same file anymore. But rsync gets it anyway. In the end rsync can't delete the file on the server because it changed (I see that in the rsync debug). I hope that explanation makes sense.

So maybe a small process flow:

1. rsync getting a file list with time-attributes (test.log.1 and test.log.2) (that's what I believe is happening, I don't know exactly how 'rsync' works)
2. the logs are rotating (test.log.1 is now named test.log.2 and .2 now .3)
3. rsync starts collecting (test.log.1 and test.log.2)
4. rsync can't delete the specific file

So is there any option that prevents that?

Answer 1

This problem is best solved at the rotation stage, not at the copy-and-archive stage. If you change the log rotation to date the files rather than give them sequential numbers, then the rsync archival doesn't need any logic to do the Right Thing™.

Assuming you are already using logrotate for rotation, then this can be done with the dateext option. But if the application is managing its own rotation, it may be harder to configure. I know that log4j can also append dates to the rotated files, but that may not be exposed through configuration and so could require a change to the application. You could also turn off application-level rotation and use logrotate instead.

Answer 2

At the end of a transfer, rsync checks that the file it has transferred matches the one is started transferring. If the files mismatch it warns you and moves on to the next one. If you have --remove-source-files it will not remove a file that it believes has been incorrectly or incompletely transferred.

Provided that there is a window of opportunity during which rsync can successfully transfer the log files, there is no great harm in repeating the rsync more frequently - or even repeating it until it succeeds:

k=3
while test 0 -lt $k && k=$((k-1))
do
    rsync .... && break
    sleep 10
done

If that cannot work for you, you're going to need to think about how you can "snapshot" the log files for long enough to copy:

ssh -q remote_host 'mkdir -p rsync.tmp; cp -p *.log rsync.tmp/'
rsync -a --remove-source-files remote_host:/path/to/logs/rsync.tmp/*.log ....