Update contents of a directory and a file that has meta about the dir safely

Question

I have a script that copies a local file to a remote directory and updates a remote file with a new line that mentions the just copied script.
Some times some of the files in the remote directory need to be deleted and the corresponding file mentioning them should be updated to delete corresponding entries as well.
Problem: The script can run from more than one instances and for more than one directory in the remote server. I.e. there are more than 1 directories in the remote server that store files that we copy based on the type passed to the instance. How can I make the copy of the file and the update of the “log” somewhat atomic?
I don’t have so much concurrent instances running for it to be a serious problem but I was wondering if there is a way to do such changes making sure the file is updated correctly.
E.g would the following work?

scp file.bin remoteserver:/foo/$type/  
grep -v "oldfile.bin" entries.log > entries.log.backup && mv entries.log.backup entries.log  
echo "$record" >> entries.log  

Answer 1

You need to use a lock; if you’re running Linux and have flock (part of util-linux), it’s as simple as

scp file.bin remoteserver:/foo/$type/
(
  flock 9
  grep -v "oldfile.bin" entries.log > entries.log.backup && mv entries.log.backup entries.log  
  echo "$record" >> entries.log
) 9>entries.log.lock

This will copy the file, wait to acquire the lock, then update the log exclusively. I’m assuming you wanted to append to the log.

The idea here is to use entries.log.lock as a lock file (we can’t use the log file itself since we’ll be replacing it). 9>entries.log.lock creates the file if necessary and associates file descriptor 9 to it, in the subshell which will run the commands between parentheses. flock 9 acquires a lock on the file pointed at by file descriptor 9, waiting if necessary; the lock will remain held until the file descriptor is closed, which will happen when the subshell exits. grep, mv and echo then proceed under the lock.

Concurrent processes will be able to scp in parallel, but the log manipulation will be serialised.

If you don’t have flock, What Unix commands can be used as a semaphore/lock? has various alternatives (the mkdir approach will work everywhere).