10

After going through the bash documentation, this question and this one it's still not clear to me how can I perform atomic write (append) operations to a file in bash. I have a script that is run in multiple instances and at some point must write data to a file:

echo "$RESULT" >> `pwd`/$TEMP_DIR/$OUT_FILE

How is it possible to make all write operations from all concurrently running scripts to that file atomic (so that data from one instance doesn't overlap data from another)?

  • 1
    FYI you don't need `pwd`; you can just use a dot (.). Also you should quote that whole file name since it includes variables. – Wildcard Apr 5 '16 at 17:51
  • 1
    You may also want to look into the possibility of using FIFOs. – Wildcard Apr 5 '16 at 17:55
  • @Wildcard Thanks. I was using pwd earlier in the script to notify the user about the current working directory and also write entries in a log file. looking over FIFOs now. – Sebi Apr 5 '16 at 17:59
  • 1
    Actually there is a better introduction to FIFOs right on this very site; the link I provided a few minutes ago is the POSIX specs for mkfifo and not exactly introductory level. – Wildcard Apr 5 '16 at 18:02
  • Whether it's a FIFO or a file, you still run the risk that two instances write at the same time and garble each other's outputs. – clacke Dec 31 '18 at 4:15
10

It seems you need to use flock as in the example from man (http://linux.die.net/man/1/flock)

(
flock -x 200

# Put here your commands that must do some writes atomically

) 200>/var/lock/mylockfile 

And put all your commands that must be atomic in ().

  • The write operation will be atomic even though ( flock -s 200 # Put here your commands that must do some writes atomically ) 200>/var/lock/mylockfile is sread across multiple concurrently running scripts? Or, conversely, is the lock only relative to /var/lock/mylockfile ? – Sebi Apr 5 '16 at 17:49
  • 1
    Your script locks /var/lock/mylockfile, but only one instance of your script can get a write lock on it. Other instances will wait untill /var/lock/mylockfile is available for locking. – Sergei Kurenkov Apr 5 '16 at 18:14
  • -s is a shared lock - suitable for reads, for a write lock you must use -x / -e. It is in the man page you linked. – Evan Benn Oct 11 '18 at 1:57
  • At what point does the unlock occur in this code? As soon as something gets written to the /var/lock/mylockfile by the process that called flock -x? – Chris Stryczynski Oct 14 '18 at 9:42
  • @ChrisStryczynski You can run this example (flock -x 200; date; sleep 10; date;) 200>/var/lock/mylockfile in two shells and see that unlocks happen only after all commands in () are executed (that is in a subshell) – Sergei Kurenkov Oct 15 '18 at 8:51
4

flock is one of the ways of interlocking operations. The utility is part of the util-linux toolset and is only available for Linux. Other utilities, available across a wider range of platforms, are based around Daniel J. Bernstein's setlock utility from his daemontools package:

These tools operate with a slightly different paradigm to that used in M. Kurenkov's answer (one that flock can also employ, but does not in that answer). One invokes the setlock program to chain load to the command that must be interlocked. setlock itself opens and locks the lock file, and leaves a file descriptor for it open in its process. The lock persists for as long as that process does (unless the subsequent command chained to explicitly releases the lock by finding and closing the open file descriptor).

For the case in the question one must interlock the command that produces the output line, being aware that this invokes an external echo in place of a shell built-in echo command:

setlock mylockfile echo "$RESULT" >> ./$TEMP_DIR/$OUT_FILE

In this case it is not necessary to interlock opening the output file in append mode. If it were, one would have to open that file within the lock, which necessitates either using programs like fdredir/redirfd:

setlock mylockfile fdredir --append 1 "./$TEMP_DIR/$OUT_FILE" echo "$RESULT"
which one can turn into a shell function if one wants:

outfile() { setlock mylockfile fdredir --append 1 "./$TEMP_DIR/$OUT_FILE" "$@" ; }
[…]
outfile echo "$RESULT"
or sticking with the shell syntax and having it interpreted by a second shell running under the interlock, requiring some non-trivial quoting if one's shell variables are not exported as environment variables:

setlock mylockfile sh -c 'echo '"$RESULT"' >> "./'$TEMP_DIR'/'$OUT_FILE'"'

This of course generalizes to things other than writing to output files:

setlock mylockfile sh -c '… interlocked ; stuff …'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.