Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Background

Here's what I want to achieve. I have a (potentially) long-running process. Now I have locking in place just fine and all is generally in order.

However, since this is a scheduled job, the script will be executed again and again and again. In the real script I have, a file keeps track of that failure count. When it reaches a threshold a mail gets sent off to the admin, in case the original script became zombie or otherwise stuck. If the threshold is reached and the initial process that claims to have the lock doesn't live anymore, the lock is broken and an alternative text is sent to the admin informing about the condition. All of this is probably not really relevant to the problem, but I want to give the complete picture here.

Whenever the threshold is reached, the failure count in the file gets reset and starts over from there. The original instance thus may or may not get to see the failure count file.

The original script instance that held the lock will (hopefully) eventually finish its job and exit.

Question

How can I implement a scheme whereby the "failed" instances that encounter the lock send a signal (I tried SIGUSR1) to the original instance and the original instance keeps track of how many times it was "pinged"?

Here's the test script I came up with so far:

#!/usr/bin/env bash
LOCKFILE=/tmp/$(basename $0).lock
let COUNTER=0

function concurrent_run
{
    let COUNTER=COUNTER+1
    echo -e "\ncaught SIGUSR1 (counter = $COUNTER)"
}

if ( set -o noclobber; echo "$$" > "$LOCKFILE" ) 2> /dev/null; then
    trap 'rm -f "$LOCKFILE"; exit $?' INT TERM EXIT
    trap concurrent_run USR1
    echo "I was first, going to sleep"
    sleep 100000
    trap - INT TERM EXIT USR1
else
    LOCKPID=$(cat "$LOCKFILE")
    echo "Lock already held by $LOCKPID"
    [[ -n "$LOCKPID" ]] && kill -0 $LOCKPID && { kill -USR1 $LOCKPID; echo "... knock knock? ($LOCKPID)"; }
    exit 1
fi
echo "End: $COUNTER"
exit 0

(the sleep is mimicking the long-running job the real script performs)

I am testing on Linux and I have tried the name for the signal as USR1 and SIGUSR1, both giving the same result (i.e. none).

builtin trap -l

gave me 10 as SIGUSR1, so I also tried a manual kill -10 PID on the PID of the original instance. Unfortunately I cannot get it to see the signal.

What am I doing wrong?


How to test

Start one instance of the script (I named it sigtest):

$ ./sigtest
I was first, going to sleep

Start a second instance of the script in another terminal, window, pane (same user!):

$ ./sigtest
Lock already held by 15360
... knock knock? (15360)

Repeat ...

Expected outcome in the first terminal (skipping empty lines):

caught SIGUSR1 (counter = 1)
caught SIGUSR1 (counter = 2)
caught SIGUSR1 (counter = 3)
caught SIGUSR1 (counter = 4)

Bash version

$ echo $BASH_VERSION
4.1.5(1)-release
share|improve this question

2 Answers 2

up vote 3 down vote accepted

man bash:

If bash is waiting for a command to complete and receives a signal for which a trap has been set, the trap will not be executed until the command completes. When bash is waiting for an asynchronous command via the wait builtin, the reception of a signal for which a trap has been set will cause the wait builtin to return immediately with an exit status greater than 128, immediately after which the trap is executed.

Make it sleep 100 instead and wait for it to finish. I don't know whether multiple signals are handled multiple times, though.

share|improve this answer
    
unfortunately this doesn't help as the counter doesn't get incremented. Any way around this? Of course a binary result is still better than none, but a numeric one would be even better. –  0xC0000022L Feb 27 at 19:36
    
Guess not, because of the blocking nature of it. Thanks for your answer. As I wrote in my previous comment, better binary than none. –  0xC0000022L Feb 27 at 19:42
    
@0xC0000022L You "just" have to execute the long-running job in the background and execute wait. Or rather: A wait loop. –  Hauke Laging Feb 27 at 19:47

You can communicate the count to the long-running job in two general ways:

  1. A shared counter protected by a second lock; or
  2. A collection of unshared counters which are aggregated by the long-running job.

The shared counter is simple enough:

if (set -o noclobber; echo $$ > /tmp/grumpy-waiters.lock); then
  echo >> /tmp/grumpy-waiters
  rm -f /tmp/grumpy-waiters.lock
fi

The long-running job can then read the counter like so:

num_grumpy_waiters=0
if [ -s /tmp/grumpy-waiters ]; then
  num_grumpy_waiters=$(wc -c < /tmp/grumpy-waiters)
fi

The unshared collection approach involves the creation of several files without need for locking. This is less risky but more elaborate. Each failed instance creates or updates its own counter:

grumpy_shard=1
if [ -s /tmp/grumpy.${LOCKPID}.$$ ]; then
  grumpy_shard=$(expr $(cat /tmp/grumpy.${LOCKPID}.$$) + 1)
fi
echo ${grumpy_shard} > /tmp/grumpy.${LOCKPID}.$$

The long-running instance can pick up these counters and aggregate them right after it releases the master lock:

num_grumpy_waiters=$(cat /tmp/grumpy.$$.* 2>/dev/null | awk 'BEGIN{s=0}NF>0{s=s+int($1)}END{print s}')
rm -f /tmp/grumpy.$$.* 2>/dev/null

In any case, using signals to interrupt the shell script won't cause a timely interruption. If you need that, then the script needs to fork a listener into its own background so that it can respond to a signal immediately.

share|improve this answer
    
Phew :) ... grumpy waiters? A reference to a Mitchell and Webb sketch, perhaps? Thanks for your answer! –  0xC0000022L Mar 1 at 10:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.