How safe it is to output to <dir> simultaneously with rm <dir>/*

Question

Sometimes I need to remove all the contents of a directory and create new files there. Can I do something like this and expect all new files to remain intact:

% rm -rf regression/* & ( sleep 10 ; run_regression )

where run_regression timestamps its output files so that they would have unique names and places them in regression?

My thinking is that the shell would resolve regression/* into an explicit list of pre-existing filenames and then rm would be removing the files on that explicit list, but not the new files that run_regression would be creating contemporaneously with rm. Since run_regression timestamps its files there should be no name clashes.

However, I'm not quite sure how to tell when the shell is done listing the files and rm starts to work. Is the above 10 sec adequate? Can I do something like this in bash:

% rm -rf regression/* & ( wait_unil_names_are_resolved ; run_regression )

Per comment clarifying that I am indeed asking whether the shell guarantees that wildcards would be expanded into filenames before invoking the tool, even if it's a tool intimately known to the shell. I can imagine that the developer of both the shell and the tool may be tempted to pipeline wildcard expansion with the tool; I hope though that there are standards preventing that.

Answer 1

Although your command probably works, here is a test case:

$ ls
$ echo * $(sleep 1)&touch file1
[1] 12798
$ file1

[1]+  Done                    echo * $(sleep 1)

Note that file1 was not typed in, it was the output of the echo command.

Edit:

Another test run:

$ ls
$ touch file1
$ for i in {1..5000}; do rm * & touch file$i; wait;done|grep file
rm: cannot remove '*': No such file or directory
***previous line repeated 14 times***

Answer 2

This is not safe.

You have not specified what the problem is that you are trying to solve. If your problem is that you want your directory to always be there but be cleaned up from time to time, I would suggest explicitly removing files older than a check file (the sleep 1 is me being paranoid):

touch regression.delete \
&& find regression \! -newer regression.delete -delete & \
&& sleep 1 \
&& run_regression

That will have problems if you have subdirectories, you could instead write

touch regression.delete \
&& find regression -mindepth 1 -maxdepth 1 \! -newer regression.delete -exec rm -rf '{}' \; & \
&& sleep 1 \
&& run_regression

If your problem is that you want to start your program as fast as possible, if the momentary absence of the directory is possible and it is not a mountpoint, I usually run something like

mkdir regression.new \
&& chmod --reference regression regression.new \
&& mv regression regression.delete \
&& mv regression.new regression \
&& rm -rf regression.delete & \
run_regression

That should allow you to start run_regression almost instantly.

Replying to your edit (and editing myself following research in another answer), wildcards must be expanded before the rm command is launched, but the crux of your problem is to know whether the expansion is done after the shell forks. POSIX spec of asynchronous execution does not explicitly specify one way or another as far as I can see, and section 2.1 certainly implies that expansion is a distinct operation and prior to actual fork/exec of the command, but testing (by @adonis, replicated by me using bash 4.3.42(1)) shows that bash takes the most efficient way: if the wildcard expansion takes time then modifications executed by the following command can well influence that expansion. Your original idea therefore risks deleting files you don't want to delete.

I looked at bash source, and execute_cmd.c explicitly states that the fork is done before word expansion:

3922 | /* If we're in a pipeline or run in the background, set DOFORK so we
3923 |  make the child early, before word expansion.  This keeps assignment
3924 |  statements from affecting the parent shell's environment when they
3925 |  should not. */

Answer 3

rm -rf regression/* runs in parallel with ( sleep 10 ; run_regression ). This means that you have no guarantee as to the order of things. rm -rf regression/* first collects the list of files in the regression directory, then invokes rm to delete them. This doesn't happen by magic, it's the shell doing the work as part of evaluating the command rm -rf regression/*, and that happens after the fork caused by the & operator. If the collection step takes less than 10 seconds, the files created by run_regression are safe. If it takes more than 10 seconds for the collection step to reach a file created by run_regression, that file will get deleted.

The deletion of the file won't actually affect run_regression, unless it closes the file and reopens it. Deleting a file doesn't affect processes that have the file open: the file keeps existing, without a directory entry (i.e. a hard link count of 0), until all processes that have it open close it. But you won't be able to access the program's output, since it'll be deleted.

So don't do this. Don't rely on timing: with such a high delay as 10 seconds, it'll work during testing (especially as there'll probably be few files, a warm cache, no I/O peak, no system suspension, etc. during your testing), but sooner or later it'll fail in production.

If you really want to keep the directory and delete the files in it, do the file name collection first.

files_to_delete=(regression/*)
rm -rf "${files_to_delete[@]}" & run_regression

(This assumes a shell with arrays. In plain sh, use set regression/*; rm -rf "$@" & run_regression.) Of course this assumes that the files run_regression only creates files that don't exist, if it overwrites existing files then those files will get deleted.

You probably don't need all this complexity: just run

rm -rf regression/*
run_regression

Unless the list of files is so huge that it doesn't fit in the cache, or unless the filesystem has unusually slow write operations, gathering the list of names is longer than deleting them, so it won't make a performance difference.

If the performance of the removal operation is really bad (which, again, would be unusual), create a new directory.

mv regression regression.old
mkdir regression
rm -rf regression.old &
run_regression

Answer 4

mv regression regression.old
rm -rf regression.old &
mkdir regression
run_regression

Rename the old regression directory, delete it in the background, make a new regression directory, and then run your program.

if run_regression creates the directory itself if it doesn't exist then the third step isn't necessary.

A safer version, in case regression.old already exists, would be to use mktemp to create and use a temporary directory in the current dir:

td=$(mktemp -d -p .)
mv regression "$td/"
rm -rf "$td" &
unset td
mkdir regression
run_regression

Answer 5

It's only safe if you use new filenames. The shell knows about filenames, not their inode, etc., and does the globbing (expansion of wildcards) before running a command. According to POSIX:

2.6.6 Pathname Expansion

After field splitting, if set -f is not in effect, each field in the resulting command line shall be expanded using the algorithm described in Pattern Matching Notation, qualified by the rules in Patterns Used for Filename Expansion.

That is, it's a well-defined step in the parsing which takes place before actually executing the command. Most of the complicated cases in POSIX deal with redirection and assignments. There are none in this example, so this is what applies:

2.9.1 Simple Commands

2. The words that are not variable assignments or redirections shall be expanded. If any fields remain following their expansion, the first field shall be considered the command name and remaining fields are the arguments for the command.

The example shown in the question makes it appear that no directories are removed. If you happen to rely upon the existence of a subdirectory which might have been removed, the same caveat applies.

Presumably your timestamp (ten seconds is different for seconds in a timestamp) would be part of the resulting filenames.