0

I have a script that copies a local file to a remote directory and updates a remote file with a new line that mentions the just copied script.
Some times some of the files in the remote directory need to be deleted and the corresponding file mentioning them should be updated to delete corresponding entries as well.
Problem: The script can run from more than one instances and for more than one directory in the remote server. I.e. there are more than 1 directories in the remote server that store files that we copy based on the type passed to the instance. How can I make the copy of the file and the update of the “log” somewhat atomic?
I don’t have so much concurrent instances running for it to be a serious problem but I was wondering if there is a way to do such changes making sure the file is updated correctly.
E.g would the following work?

scp file.bin remoteserver:/foo/$type/  
grep -v "oldfile.bin" entries.log > entries.log.backup && mv entries.log.backup entries.log  
echo "$record" >> entries.log  
2

You need to use a lock; if you’re running Linux and have flock (part of util-linux), it’s as simple as

scp file.bin remoteserver:/foo/$type/
(
  flock 9
  grep -v "oldfile.bin" entries.log > entries.log.backup && mv entries.log.backup entries.log  
  echo "$record" >> entries.log
) 9>entries.log.lock

This will copy the file, wait to acquire the lock, then update the log exclusively. I’m assuming you wanted to append to the log.

The idea here is to use entries.log.lock as a lock file (we can’t use the log file itself since we’ll be replacing it). 9>entries.log.lock creates the file if necessary and associates file descriptor 9 to it, in the subshell which will run the commands between parentheses. flock 9 acquires a lock on the file pointed at by file descriptor 9, waiting if necessary; the lock will remain held until the file descriptor is closed, which will happen when the subshell exits. grep, mv and echo then proceed under the lock.

Concurrent processes will be able to scp in parallel, but the log manipulation will be serialised.

If you don’t have flock, What Unix commands can be used as a semaphore/lock? has various alternatives (the mkdir approach will work everywhere).

11
  • Yes append. But could you explain a bit the syntax and what's going on? What is 9>entries.log.lock?
    – Jim
    Oct 18 '17 at 8:05
  • Also is it possible the copy and the update to be both atomic? I.e. other instances wait for that copy & update to finish
    – Jim
    Oct 18 '17 at 8:12
  • If you want the copy to be atomic, you can move the scp inside the subshell, after flock. In that case you could perhaps use the last pattern given in the manpage instead, to lock the whole shell script. Oct 18 '17 at 8:13
  • Where does the 9 for file descriptor come from? Could I end up not being able to use that number? Is it arbitrary?
    – Jim
    Oct 18 '17 at 8:20
  • Also the last pattern you are referring is it the section: This is useful boilerplate code for shell scripts. Put it at the top of the shell script you want to lock...
    – Jim
    Oct 18 '17 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.