2

I am using fswatch and xargs to watch for file system changes, and run commands depending on which files are changing. fswatch prints the changed files to stdout, and xargs <my command> consumes them from stdin.

If I use xargs -n1, my command will run after every change. If I use xargs -n100, my command won't run until there have been 100 changes. I'd like to batch all of the changes (arguments) that occur within 1 second.

If xargs receives 50 lines on stdin at once, I'd like xargs to pass all 50 of them to my command in a single invocation, rather than 50 separate invocations as with xargs -n1. But if xargs only receives 1 line per minute, I don't want to wait 50 minutes before my command is invoked; I want it to be relatively interactive (i.e. ~1 second maximum, provided that xargs has received a nonzero number of arguments on stdin).

Thanks in advance!

  • 1
    Note, fswatch has a --batch-marker option that puts a special NoOp line in the output after a "burst". – meuh Aug 17 '18 at 15:04
  • @meuh I think this is the best answer for me. – arya Aug 18 '18 at 18:52
4

If you can use Bash, you could use the -t (timeout) option for read, and/or the SECONDS variable.

This is a bit ugly but works to group input that comes roughly within a second:

#!/bin/bash

while true; do 
        values=()
        start=$SECONDS
        while [ "$SECONDS" -lt $((start + 1)) ]; do
                if IFS= read -r -t 1 line; then
                        values+=("$line")
                fi
        done
        if [ "${#values[@]}" != 0 ]; then
                echo got "${values[@]}"
                # do something useful with "${values[@]}" here 
        fi
done

The timeout on read keeps it from blocking forever, and the test against SECONDS makes sure that the inner loop is broken even in the pathological case where read gets input once per second, for ever.

1

Use Lsyncd instead of reinventing the wheel;

lsyncd -rsyncssh /home remotehost.org backup-home/

https://github.com/axkibe/lsyncd/blob/master/README.md

Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes.

  • This answer would be a lot more useful if you added an example of how to use it (and maybe a link to the software itself). – ilkkachu Aug 17 '18 at 0:07
  • I’m not sure how this relates to my question, as I wasn’t asking how to sync files. Could you clarify? – arya Dec 28 '18 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.