0

I run a Sheevaplug (small ARM server) with Debian 9. It does not have any third-party repos enabled in sources.list / sources.list.d.

I have a backup script which runs as root, and uses at. I think something broke on Sep 13, because I am getting these emails that look like they come from at. They are daily, like my backups. The body of the message just says Killed.

I can't think what would be sending SIGKILL to my process! Without gathering any more information than I have now, can you think of any reason this would happen?

It can't be from the OOM killer (Out of Memory condition), because I have a full kernel log in dmesg which does not show any OOM messages.

The at job is

#!/bin/sh
# at uses sh shell

set -e                   
cd /d/backup/jenkins-desktop/

for i in */; do                              
    nice ionice -c 3 rdiff-backup "$i" ../jenkins-desktop.rdiff/"$i"
done

I doubt it's systemd SystemCallFilter=, and that would send SIGSYS by default. I see that a couple of rlimits send SIGKILL. But I'm not doing anything to set rlimits myself; also it looks like in both cases you would be killed by SIGXCPU first, which defaults to fatal and should show "CPU time limit exceeded".

I have looked in journalctl --since=-2d -p notice and there are no errors, only some success messages from anacron.


Return-path: <root@brick>
Envelope-to: root@brick
Delivery-date: Thu, 13 Sep 2018 02:14:15 +0100
Received: from root by brick with local (Exim 4.89)
        (envelope-from <root@brick>)
        id 1g0GD0-0000Xr-Bz
        for root@brick; Thu, 13 Sep 2018 02:14:14 +0100
Subject: Output from your job     1843
To: root@brick
Message-Id: <E1g0GD0-0000Xr-Bz@brick>
From: root <root@brick>
Date: Thu, 13 Sep 2018 02:14:14 +0100
X-IMAPbase: 1541805998 113
Status: O
X-UID: 1

Killed
0

The body of the message just says Killed.

Sorry, this was incorrect.

The body of the first message says Killed. I think this was a one-off killing performed by an admin (me) :-).

The reason I am getting daily messages can be investigated by looking at the subsequent messages. Or, I should be careful now and say the second and last messages look the same :-).

Previous backup seems to have failed, regressing destination now.
Exception '[Errno 28] No space left on device' raised of class '<type 'exceptions.IOError'>':
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/robust.py", line 32, in check_common_error
    try: return function(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/restore.py", line 468, in get_fp
    Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Rdiff.py", line 73, in write_patched_fp
    rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp), out_fp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 64, in copyfileobj
    outputfp.write(inbuf)

Exception '[Errno 28] No space left on device' raised of class '<type 'exceptions.IOError'>':
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])

You might wonder "regressing destination" seems to fail with "No space left on device". I'm not sure, because there seems to be a fair amount of space on the drive, but that's a question for another day.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.