RSync Error in script

Question

At my job, we have a central Backup server, running Debian Wheezy, and on-site servers at each site, also running Debian Wheezy.

A few weeks ago, the central office tech emailed me that the backup did not complete properly the night before. We have been troubleshooting ever since, but still cannot seem to solve the issue. The only thing spat back was the following in a cron email:

rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [generator=3.0.9]
rsync error: received SIGUSR1 (code 19) at main.c(1316) [receiver=3.0.9]

Googling that phrase leads to pretty much nothing. I found a post from 2002 about removing the -v switch, but that's not being used in the script. The script that runs every night is below:

#!/bin/sh
set -e
x="delete --exclude-from=r_filter --delete-excluded"

rsync -aq --$x site1.company.com:/etc  /BACKUPS/site1
rsync -aq --$x site1.company.com:/home /BACKUPS/site1

It is set to run at 3:00am, Monday to Friday, from the central backup server. If they try running it manually during the day, it runs fine (cause most of the files were backed up previously?). It's using the -a switch, so I assume it can archive opened files? That's about all I can think of.

What would be our next step for figuring this out?

Answer 1

If something happens when you run a job in a crontab at a certain time and it doesn't happen if you make a test where you run the job in one minute, there are two likely possibilities:

• There's another process on a crontab that somehow interferes with yours.
• There's a human process going on at that time, such as the cleaning staff unplugging a computer to plug the vacuum cleaner.

Your rsync process is receiving signals at a certain time of the night. The first thing I'd look for is another process in a crontab sending signals that it shouldn't be sending.

(If something runs fine from the command line but fails when run from cron, that's a whole different kettle of fish.)

Answer 2

You can try making your commands immune to the signals in case it happens again, and at the same time in the background catch the signal if it is being directed at your shell through some parent in the same process group. Eg:

#!/bin/bash
( trap 'echo got signal; date; ps ax; kill $pid; exit' sigint sigterm sighup
  sleep 999999 &
  pid=$!
  wait
) &

trap '' sigint sigterm sighup
rsync ...
rsync ...
kill -hup $!

The trap '' will ignore the 3 listed signals on following commands. The background part in () will trap the signals and, for example, do a ps or something similar to perhaps find what is running at this time.

I'd look for something stupid like an over-enthusiastic logrotate command doing this signalling.