5

I have a long running (3 hours) shell script running on a CentOS 7 machine. The script runs a loop with an inner loop and calls curl in each iteration.

I'm starting the script with PM2 because it's already on the system and it's good for managing processes. However it seems that it might not be good for shell scripts. When I came in this morning I saw that PM2 had restarted my shell script 6 times. The PM2 logs say it received a SIGINT and was restarted. Since this script results in data being pushed to a database that means my data has been pushed 6 times. That's no bueno.

I'm the only person that logs into the box so it's not another user.

So, next question is whether this is a bug in PM2 or a legit SIGINT. Which begs the question: if it is legit, where is it coming from? I have to determine (if possible) if the OS is somehow killing this process before I submit this as a bug in PM2 (which seems like the most likely thing).

  • Did you track where this signal came from? I'm experiencing the same with pm2 and nodejs app. – leafnode Aug 22 '17 at 10:31
  • No, this stopped happening when I put my node processes in docker. I think it was an issue between PM2 and CentOS. – jcollum Aug 22 '17 at 19:36
7

sysdig can monitor for these using the evt.type=kill filter:

# terminal uno
perl -E 'warn "$$\n"; $SIG{INT}= sub { die "aaaaargh" }; sleep 999'

# terminal dos
sysdig -p '%proc.pname[%proc.ppid]: %proc.name -> %evt.type(%evt.args)' evt.type=kill

# terminal tres
kill -INT 11943  # or whatever

A more specific filter may be necessary to avoid e.g. systemd spam from cluttering up the sysdig output, or grep for your process names or pids:

# sysdig -p '%proc.pname[%proc.ppid]: %proc.name -> %evt.type(%evt.args)' evt.type=kill
systemd[1]: systemd-udevd -> kill(pid=11969(systemd-udevd) sig=15(SIGTERM) )
systemd[1]: systemd-udevd -> kill(res=0 )
systemd[1]: systemd-udevd -> kill(pid=11970(systemd-udevd) sig=15(SIGTERM) )
systemd[1]: systemd-udevd -> kill(res=0 )
systemd[1]: systemd-udevd -> kill(pid=11971(systemd-udevd) sig=15(SIGTERM) )
systemd[1]: systemd-udevd -> kill(res=0 )
sshd[11945]: bash -> kill(pid=11943(perl) sig=2(SIGINT) )
sshd[11945]: bash -> kill(res=0 )

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.