5

Sometimes I need to remove all the contents of a directory and create new files there. Can I do something like this and expect all new files to remain intact:

% rm -rf regression/* & ( sleep 10 ; run_regression )

where run_regression timestamps its output files so that they would have unique names and places them in regression?

My thinking is that the shell would resolve regression/* into an explicit list of pre-existing filenames and then rm would be removing the files on that explicit list, but not the new files that run_regression would be creating contemporaneously with rm. Since run_regression timestamps its files there should be no name clashes.

However, I'm not quite sure how to tell when the shell is done listing the files and rm starts to work. Is the above 10 sec adequate? Can I do something like this in bash:

% rm -rf regression/* & ( wait_unil_names_are_resolved ; run_regression )

Per comment clarifying that I am indeed asking whether the shell guarantees that wildcards would be expanded into filenames before invoking the tool, even if it's a tool intimately known to the shell. I can imagine that the developer of both the shell and the tool may be tempted to pipeline wildcard expansion with the tool; I hope though that there are standards preventing that.

  • @John1024: fixed the typo. – Michael May 31 '16 at 21:18
4

This is not safe.

You have not specified what the problem is that you are trying to solve. If your problem is that you want your directory to always be there but be cleaned up from time to time, I would suggest explicitly removing files older than a check file (the sleep 1 is me being paranoid):

touch regression.delete \
&& find regression \! -newer regression.delete -delete & \
&& sleep 1 \
&& run_regression

That will have problems if you have subdirectories, you could instead write

touch regression.delete \
&& find regression -mindepth 1 -maxdepth 1 \! -newer regression.delete -exec rm -rf '{}' \; & \
&& sleep 1 \
&& run_regression

If your problem is that you want to start your program as fast as possible, if the momentary absence of the directory is possible and it is not a mountpoint, I usually run something like

mkdir regression.new \
&& chmod --reference regression regression.new \
&& mv regression regression.delete \
&& mv regression.new regression \
&& rm -rf regression.delete & \
run_regression

That should allow you to start run_regression almost instantly.

Replying to your edit (and editing myself following research in another answer), wildcards must be expanded before the rm command is launched, but the crux of your problem is to know whether the expansion is done after the shell forks. POSIX spec of asynchronous execution does not explicitly specify one way or another as far as I can see, and section 2.1 certainly implies that expansion is a distinct operation and prior to actual fork/exec of the command, but testing (by @adonis, replicated by me using bash 4.3.42(1)) shows that bash takes the most efficient way: if the wildcard expansion takes time then modifications executed by the following command can well influence that expansion. Your original idea therefore risks deleting files you don't want to delete.

I looked at bash source, and execute_cmd.c explicitly states that the fork is done before word expansion:

3922 | /* If we're in a pipeline or run in the background, set DOFORK so we
3923 |  make the child early, before word expansion.  This keeps assignment
3924 |  statements from affecting the parent shell's environment when they
3925 |  should not. */
  • 2
    Oh, and I just thought of this: when I have this type of problem I usually organize the files in one (timestamped) subdirectory per run. That way it is easy to keep (say) the results of the last ten runs. I usually make a symbolic link called last that is created by ln -sf $timestamp last once the run finishes successfully. That way last ALWAYS points to the last run completed successfully. – Law29 May 31 '16 at 22:07
  • 3
    It is certain that the wildcard expansion happens after the shell forks. If wildcard expansion gets delayed due to a very slow filesystem, the other process is still supposed to run. So the wildcard expansion runs in parallel with starting the other process. Often wildcard expansion is faster than the initialization of a large program, but you can't rely on this, and it's likely to work better while you're testing (because the directory listing is in cache) than in real-world situations. – Gilles May 31 '16 at 22:31
  • 1
    This answer looks good to me. What's with the downvote? – John1024 Jun 1 '16 at 2:00
5

Although your command probably works, here is a test case:

$ ls
$ echo * $(sleep 1)&touch file1
[1] 12798
$ file1

[1]+  Done                    echo * $(sleep 1)

Note that file1 was not typed in, it was the output of the echo command.

Edit:

Another test run:

$ ls
$ touch file1
$ for i in {1..5000}; do rm * & touch file$i; wait;done|grep file
rm: cannot remove '*': No such file or directory
***previous line repeated 14 times***
  • In other words, this proves that a command can be put into background before the pathname expansion is performed. – John1024 May 31 '16 at 22:20
  • The subshell that is spawned to run the command goes to the background. My take on this is that there is no guaranty that in command1 & command2 there is no guarantee that pathname expansion of command1 takes place before command2 gets executed – adonis May 31 '16 at 22:59
  • Yes, very good work. – John1024 May 31 '16 at 23:02
2

rm -rf regression/* runs in parallel with ( sleep 10 ; run_regression ). This means that you have no guarantee as to the order of things. rm -rf regression/* first collects the list of files in the regression directory, then invokes rm to delete them. This doesn't happen by magic, it's the shell doing the work as part of evaluating the command rm -rf regression/*, and that happens after the fork caused by the & operator. If the collection step takes less than 10 seconds, the files created by run_regression are safe. If it takes more than 10 seconds for the collection step to reach a file created by run_regression, that file will get deleted.

The deletion of the file won't actually affect run_regression, unless it closes the file and reopens it. Deleting a file doesn't affect processes that have the file open: the file keeps existing, without a directory entry (i.e. a hard link count of 0), until all processes that have it open close it. But you won't be able to access the program's output, since it'll be deleted.

So don't do this. Don't rely on timing: with such a high delay as 10 seconds, it'll work during testing (especially as there'll probably be few files, a warm cache, no I/O peak, no system suspension, etc. during your testing), but sooner or later it'll fail in production.

If you really want to keep the directory and delete the files in it, do the file name collection first.

files_to_delete=(regression/*)
rm -rf "${files_to_delete[@]}" & run_regression

(This assumes a shell with arrays. In plain sh, use set regression/*; rm -rf "$@" & run_regression.) Of course this assumes that the files run_regression only creates files that don't exist, if it overwrites existing files then those files will get deleted.

You probably don't need all this complexity: just run

rm -rf regression/*
run_regression

Unless the list of files is so huge that it doesn't fit in the cache, or unless the filesystem has unusually slow write operations, gathering the list of names is longer than deleting them, so it won't make a performance difference.

If the performance of the removal operation is really bad (which, again, would be unusual), create a new directory.

mv regression regression.old
mkdir regression
rm -rf regression.old &
run_regression
  • "[globbing] happens after the fork caused by the & operator." - are you sure? That's very surprising to me. – Random832 Jun 1 '16 at 2:46
  • @Random832 see the test by adonis and my foray into bash source: yes, globbing happens after the fork. That can be surprising, but considering that it is more efficient, maybe it should not be surprising. – Law29 Jun 1 '16 at 6:01
  • @Random832 Yes, I'm sure, and it can't be any other way due to the basics of how the shell works. Consider echo * & …. Then consider { echo *; } & …: the braces don't change the order of evaluation, they only affect the parsing. Now consider { true; echo *; } & …: adding true; doesn't affect when the wildcard expansion happens. Now consider { touch foo; echo *; } & …: the wildcard expansion has to happen after touch foo runs, it obviously will include foo. And all of this is happening in the background process. – Gilles Jun 1 '16 at 14:14
  • See, the idea that the braces don't change the evaluation wasn't clear to me. I thought that it was something like "a command can be terminated by either a semicolon or an ampersand", and that trying to run a brace-subexpression in the background either implicitly created a subshell (how can it do anything else with your last example?) or was an error. – Random832 Jun 1 '16 at 15:02
  • @Random832 The ampersand creates the subshell. The braces are merely syntax to get a sequence of commands to be treated as a single unit for parsing purposes, so that the & operator applies to the whole braced group and not just to the preceding pipeline. – Gilles Jun 1 '16 at 15:06
1
mv regression regression.old
rm -rf regression.old &
mkdir regression
run_regression

Rename the old regression directory, delete it in the background, make a new regression directory, and then run your program.

if run_regression creates the directory itself if it doesn't exist then the third step isn't necessary.

A safer version, in case regression.old already exists, would be to use mktemp to create and use a temporary directory in the current dir:

td=$(mktemp -d -p .)
mv regression "$td/"
rm -rf "$td" &
unset td
mkdir regression
run_regression
0

It's only safe if you use new filenames. The shell knows about filenames, not their inode, etc., and does the globbing (expansion of wildcards) before running a command. According to POSIX:

2.6.6 Pathname Expansion

After field splitting, if set -f is not in effect, each field in the resulting command line shall be expanded using the algorithm described in Pattern Matching Notation, qualified by the rules in Patterns Used for Filename Expansion.

That is, it's a well-defined step in the parsing which takes place before actually executing the command. Most of the complicated cases in POSIX deal with redirection and assignments. There are none in this example, so this is what applies:

2.9.1 Simple Commands

  1. The words that are not variable assignments or redirections shall be expanded. If any fields remain following their expansion, the first field shall be considered the command name and remaining fields are the arguments for the command.

The example shown in the question makes it appear that no directories are removed. If you happen to rely upon the existence of a subdirectory which might have been removed, the same caveat applies.

Presumably your timestamp (ten seconds is different for seconds in a timestamp) would be part of the resulting filenames.

  • Unless OP limits/clarifies the question, there's nothing much to expand upon. – Thomas Dickey May 31 '16 at 21:09
  • IMHO, even if you use new filenames, there is no guarantee of safety – fpmurphy May 31 '16 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.