8

I have a service which is sporadically publishing content in a certain server-side directory via rsync. When this happens I would like to trigger the execution of a server-side procedure.

Thanks to the inotifywait command it is fairly easy to monitor a file or directory for changes. I would like however to be notified only once for every burst of modifications, since the post-upload procedure is heavy, and don't want to execute it for each modified file.

It should not be a huge effort to come up with some hack based on the event timestamp… I believe however this is a quite common problem. I was not able to find anything useful though.

Is there some clever command which can figure out a burst? I was thinking of something I can use in this way:

inotifywait -m "$dir" $opts | detect_burst --execute "$post_upload"
2
  • 1
    If possible you could modify the remote rsync to add an option like ` -M --log-file=/some/path/log` which would make the local rsync keep a log of the changes in a local file you can wait for the close of.
    – meuh
    Commented Sep 7, 2017 at 17:43
  • @meuh I'm working through ssh, so there's no remote rsync. But maybe your suggestion could be the way to go otherwise.
    – Dacav
    Commented Sep 11, 2017 at 9:41

4 Answers 4

3

Drawing on your own answer, if you want to use the shell read you could take advantage of the -t timeout option, which sets the return code to >128 if there is a timeout. Eg your burst script can become, loosely:

interval=$1; shift
while :
do  if read -t $interval
    then    echo "$REPLY"            # not timeout
    else    [ $? -lt 128 ] && exit   # eof
            "$@"
            read || exit    # blocking read infinite timeout
            echo "$REPLY"
    fi
done

You may want to start with an initial blocking read to avoid detecting an end of burst at the start.

2
  • Ah! This seems really interesting! I'm totally trying it out asap
    – Dacav
    Commented Sep 11, 2017 at 12:50
  • Interestingly, on zsh it returns 1 both upon timeout or if I'm reading from /dev/null. I'm going to use this on bash, so that's not a problem, but it's a good thing to know :)
    – Dacav
    Commented Sep 11, 2017 at 12:59
3

OP here. A possible solution hack could be similar to

inotifywait -m "$dir" -e moved_to --timefmt='%s' --format '%T' | stdbuf -oL uniq | ...

EDIT: See my other answer to this question, since the second take is IMHO a better one

It works by having the time since Epoch printed at every move_to is performed, which is the final step of a single-file transfer.

This works decently, even if it could trigger the post-upload procedure multiple times if one upload lasts more than one second.

A different granularity could be obtained by changing the --timefmt flag to some different value.

Posting it here as semi-decent idea, even though I don't like it much ...still I thought of sharing it.

2
  • Better make that … | stdbuf -oL uniq | … or you'll have some large pauses due to buffering. Commented Sep 8, 2017 at 21:44
  • I came out with another solution which is a bit better than this one. See my other answer.
    – Dacav
    Commented Sep 11, 2017 at 9:58
2

Op here.

The solution I came out with: it is a script named burst:

inotifywait -e moved_to "$monitored_dir" -m \
    | burst 2 'echo run post-upload'

The burst script:

#!/bin/bash

help() {
    >&2 echo "Usage: $0 <interval> <command>"
}

set -e
trap help EXIT
interval=${1:?missing interval}; shift
: ${1:?missing command}

trap - EXIT
set +e
exec 3> >(sed "s/^/burst: /" >&2)
while read line; do
    echo "$line" >&3
    test -n "$!" && kill -term $! 2>/dev/null
    (sleep $interval && $SHELL -c "$*") &
done

Basically this is creating a subshell which waits a defined amount of time before launching the actual command ((sleep $interval && $SHELL -c "$*"). Any new line read by inotifywait will simply kill the such shell (if any exists) and create it again.

Once the burst of lines is over, the shell will be able to end the sleep, and the command will be executed.

It has some drawbacks:

  1. It kills and spawns one process for each line in stdin. Which means you want to reduce the number of lines first off (hence the -e moved_to filter in inotifywait). It could not fit w.r.t performance if the number of uploaded files is large enough!

  2. If a second burst comes in after $interval seconds (or a file transfer takes that time) chances are it will kill the running post-update process. As long as such process is idempotent or transacted this will be fine… so heads up.

0

You can make a loop that checks the current time and skips events that are identical (or more generally, similar in a way you can define) to a previous event. In bash, ksh or zsh, the following simple loop skips repeated events in a 10-second period:

inotifywait -m … | {
  previous_SECONDS=0
  previous_event=
  while IFS= read -r event; do
    if [[ "$event" = "$previous_event" && $SECONDS <= $previous_SECONDS + 10 ]]; then
      continue
    fi
    previous_SECONDS=$SECONDS previous_event=$event
    "$post_upload" "$event"
  done
}

Shells are not good at high-performance text processing, so it may be better to use a different tool. You can use awk, which for historical reasons has a very strange way to expose the time (as number of seconds since the epoch): calling srand() returns the time of the previous call to srand().

inotifywait -m … | awk '
    {srand(); current_time = srand()}
    $0 == previous_event && current_time <= previous_time + 10 {
        system(ENVIRON["post_upload"])
    }
    {previous_event = $0; previous_time = current_time}
'

Unless you really need the portability, I'd scrap this and use a more advanced scripting language such as Perl, Python or Ruby. These languages have an interface to inotify, so no need to call inotifywait and do any text parsing.

1
  • Yep, using a scripting language is also a possible solution. Also the srand story is crazy enough... my face when…
    – Dacav
    Commented Sep 11, 2017 at 9:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .