rsync error code 20 when ran under systemd

Question

Running CentOS7.4 , rsync v3.1.2

I've a python script(v3.6) that polls a remote file server location using rsync. Rsync ensures that the drop folder on the remote location is identical to the folder on my local machine. I need these folders to be identical and rsync seems up to the task.

However, whenever there are more than 40+ files, rsync errors with the following output: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(638) [sender=3.1.2] and rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(504) [generator=3.1.2]. The files are almost always a series of jpeg images with a single xml file.

I've ran the exact rsync command in a terminal shell and there were no issues. It seems that because i've daemonized my script and leave it to systemd to handle the program, it doesn't like that rsync is copying that many files. I am unsure as to the cause. I was under the impression that the Kernel's OOM killer is the culprit as rsync is receiving a SIGINT, SIGTERM or SIGHUP as stated in the error but i ran journalctl -k to spot any OOM kills and none appear relating to rsync.

My workstation has 64 GB of ram to play with and doesn't regularly use nearly all that space so I am perplexed as to what is causing rsync to do this. If more information is required please do not hesitate to ask as I need a solution for this by the end of the week.

rsync command :rsync -a -vv -Ph --temp-dir=/tmp /inbox/loc/ /outbox/loc

Python Call: retval = subprocess.check_call(['rsync','-a', '-vv', '-Ph', '--temp-dir=/tmp', inbox, outbox])

Systemd Unit File:

[Unit]
Description = CAR daemon

[Service]
Type = simple
PIDFile = /var/run/card/card.pid
ExecStart = /usr/bin/python3.6 /scratch/ssd/jamal/dev/github/CAR_WF/card/main.py

[Install]
WantedBy = multi-user.target

Journalctl sample output for one of the runs ( redacted for sensitive data ):

Dec 18 08:59:13 hostname systemd[1]: Started CAR daemon.
Dec 18 08:59:13 hostname systemd[1]: Starting CAR daemon...
Dec 18 08:59:14 hostname python3.6[2928]: sending incremental file list
Dec 18 08:59:14 hostname python3.6[2928]: delta-transmission disabled for local transfer or --whole-file
Dec 18 08:59:14 hostname python3.6[2928]: ./
Dec 18 08:59:14 hostname python3.6[2928]: FOR VFX DOCUMENT/
Dec 18 08:59:14 hostname python3.6[2928]: FOR VFX DOCUMENT/0000 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [102B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0001 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [102B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0002 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [102B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0003 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [102B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0004 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [102B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0005 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [102B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0006 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [102B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0007 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [102B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0008 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [102B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0009 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [103B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0010 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [103B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0011 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: [103B blob data]
Dec 18 08:59:15 hostname python3.6[2928]: FOR VFX DOCUMENT/0012 file.jpg
Dec 18 08:59:15 hostname python3.6[2928]: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(638) [sender=3.1.2]
Dec 18 08:59:15 hostname python3.6[2928]: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(504) [generator=3.1.2]

Output for ulimit -a:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256628
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 256628
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I should add that rsync is called as follows:

1. Systemd spawns the python process.
2. The Python script spawns a polling process upon initialisation using subprocess.Popen() with the following arguments: ['python3.6', '/path/to/poller.py', '/inbox/dir/', '/outbox/dir']
3. When the poller timer hits zero, it calls rsync to check for changes.

I hope that clears things up about where the rsync process is called from.

Answer 1

For me rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(638) was resolved by copying ssh key of the user which runs rsync to destination server. For that I ran:

ssh-copy-id [user]@[host-name]

If you don't have the key generated or the host added to known_hosts, run following first:

ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
ssh-keyscan -H [host-name] >> ~/.ssh/known_hosts

It's on Oracle Linux 7, rsync version 3.1.2