Wait for move directory to be completed before attempting to delete it

Question

I have a process running on one computer that spawns simulations by writing the simulation data to directory pre/id. Worker processes then copy a simulation from pre to a local disk, which can be on a different computer. pre is in a volume mounted with nfs. This part works well.

When a simulation is done, the results are moved to the directory result/id, which is what is causing trouble. The supervising process can decide to keep such a directory or to delete it. Occasionally, when it tries to delete result/id, the move operation seems to be incomplete, and removing the directory fails.

Everything runs on a variety of linux flavors. The workers move directories around using mv and then touch result/id/done to signal to the supervising process that the result can be read (and deleted). The supervising process uses boost::filesystem::remove_all to delete result/id.

How can I reliably wait for the move operation to be completed, before attempting to delete it?

---

Added: This code moves the result directory to where the supervising process waits for it:

mv $tempDir $finishedCasesDir # copy case to result directory
touch $finishedCasesDir/$caseName/done

This is the C++ code that waits for done to appear:

if(is_regular_file(resultPath/"done"))
{
  // get relevant result data
  ...
  // remove result directory
  remove_all(resultPath);
}

And the error:

terminate called after throwing an instance of 'boost::filesystem3::filesystem_error'
what():  boost::filesystem::remove: Directory not empty: "results/711a35ed-818e-4084-ab43-47531fdd8d11"

Answer 1

Have you come across the flock command?

It provides you a file lock within the filesystem so can be used in shell scripts.

--- edit

After my initial answer above, further edits were made to original post, to which I added comments and a final suggestion of a race condition across several machines which are using nfs, and devised a scenario. This scenario was challenged by @alexis, to which I thought it deserved a reply.

@alexis you are correct when working within one filesystem, but the situation becomes more complicated when nfs mounted file systems are involved.

It is unclear from OP exactly what mix of machines/servers/clients nfs versions are involved, but I thought it was enough to say, "You need a better syncing mechanism than "touch - rm" and indeed, OP suggests it sort of works but has a 1 in 15k chance of failure. So I suggested either, find a better way to sync or code around it.

After a little investigation on the subject, I have found some references that do show "flaws" in nfs that indicate that removing a file does not work as expected across nfs. More over, there is are differences between nfs v3 and v4, specifically to address this flaw, also nfs4 could working differently, but doesn't or else it would break compatibility with older clients.

This nfs document summarises the situation it describes the silly rename that was introduced to code around the problem, and rfc 5661 NFS 4.1 provides further detail.

--edit 2

Extract of one paragraph from above references :

Because of the design of the NFS protocol, there is no way for a file to be deleted from the name space but still remain in use by an application. Thus NFS clients have to emulate this using what already exists in the protocol. If an open file is unlinked, an NFS client renames it to a special name that looks like ".nfsXXXXX". This "hides" the file while it remains in use. This is known as a "silly rename." Note that NFS servers have nothing to do with this behavior.

Answer 2

Idea #1 - alternative approach?

Rather than touch a file what if you waited for the mv process to complete instead.

$ mv $tempDir $finishedCasesDir & # copy case to result directory
$ wait %1 && touch $finishedCasesDir/$caseName/done

This will only touch the file when the mv process has terminated.

Example

Here's an example using the sleep command as a stand in for your mv command.

start time

$ date
Thu Feb 13 21:23:33 EST 2014

start simulated "mv" command

$ sleep 10 &
[1] 28561

we then "wait" for it to finish

$ wait %1 && echo 'all done!'
[1]+  Done                    sleep 10
all done!

confirming that we were waiting for ~10 secs.

$ date
Thu Feb 13 21:23:48 EST 2014

continue

$ ...boost program can then run...

Idea #2 - NFS issue?

Based on feedback from @Gilles, I hadn't realized you were working with these files over NFS. I believe Gilles is 100% correct, I too have encountered similar issues when working with files over NFS where a process may still have access to a NFS mounted directory that you're attempting to delete. When you do this NFS will typically create a .nfsXXXX file in the directory, which will foil your Boost applications attempts to delete the file since it's effectively not empty.

NOTE: Having a shell whose CWD (Current Working Directory) is a sub-directory within this NFS mount is enough to cause this issue.

You can read more about this issue here in this article, titled: What is this .nfs file and why can I not remove it?.

excerpt from above article

% echo test> foo
% tail -f foo
test
^Z
Suspended
% rm foo
% ls -A
.nfsB23D
% rm .nfsB23D
% ls -A
.nfsC23D
% lsof .nfsC23D
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
tail    1257 robh    0r  VREG  176,6        5 3000753 .nfsC23D
%

Notice: you can use the tool lsof to determine what process is maintaining a file descriptor.

References

• Can I somehow add a “&& prog2” to an already running prog1?