One might think that
echo foo >a cat a | rev >a
oof; but instead it is left empty.
- How would one otherwise apply
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.Sign up to join this community
There's an app for that! The
sponge command from
moreutils is designed for precisely this. If you are running Linux, it is likely already installed, if not search your operating system's repositories for
moreutils. Then, you can do:
echo foo >a cat a | rev | sponge a
Or, avoiding the UUoC:
rev a | sponge a
The reason for this behavior is down to the order in which your commands are run. The
> a is actually the very first thing executed and
> file empties the file. For example:
$ echo "foo" > file $ cat file foo $ > file $ cat file $
So, when you run
cat a | rev >a what actually happens is that the
> a is run first, emptying the file, so when the
cat a is executed the file is already empty. This is precisely why
sponge was written (from
man sponge, emphasis mine):
sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before writing the output file. This allows constructing pipelines that read from and write to the same file.
another way to fix this is to use a writing method that does not truncate
rev a | dd conv=notrunc of=a
this only works because:
rev reads content before producing output and the output is never longer than the amount already read
the new file content is same size or larger than the original (in this case same size)
This approach may be useful for in-place modification of files too large to keep temporary copies of.
cat a | rev > a
In the pipeline above, the shell forks two subprocesses, one for each of the two parts of the pipeline. Those subprocesses then run the commands in question, first processing any redirections, then calling one of the
exec*() functions to start the external utility. The subprocesses run in parallel, and there are no timing guarantees between them.
Exec'ing a process isn't very fast, so usually what happens is that the shell on the right-hand side manages to set up the redirection before
cat has a chance to read the file. The output redirection
> a truncates the file, so
cat has nothing to read,
rev receives no data and produces no data. Even if you used a redirection the left-hand side too (
cat < a | rev > a),
a might get opened for reading before it was truncated, but
cat still probably wouldn't have time to actually read it before that.
On the other hand, this quite consistently prints
a contains: foo on my system:
echo foo > a; cat < a | tee a > /dev/null ; echo "a contains: $(cat a)"
tee that truncates the file, so this happens after the
cat has a better chance of having time to read the file. But if the file was large enough, it might get truncated in the middle of reading it.
I said might and probably there, because indeed the exact opposite can happen, if the OS decides to schedule the processes in another fashion.
How would one otherwise apply
The usual solution is to use a temporary file:
cat a | rev > b && mv b a
Though there's the usual problem of possibly overwriting an existing file, unless you can be sure that the temporary file name is available. You should probably use
f=$(mktemp ./tmp.XXXXXX) cat a | rev > "$f" && mv "$f" a || rm "$f"
Alternatively, you can use the
sponge tool, which makes sure to read all of the input it gets before opening the output file (otherwise it's like
cat a | rev | sponge a
rev < a | sponge a
sponge > a would be a mistake for the same reason the original command doesn't work.
Sponge is from moreutils, and not a standard tool. Some alternatives for it are listed in Completely buffer command output before piping to another command?
Some utilities may implement a similar feature themselves, e.g.
sort -o outputfile only opens the output file after finishing, see Does sort support sorting a file in-place, like `sed --in-place`?
>file create a new empty file or truncates an existing file. As such, there's nothing left in the file for
rev to read.
As other answers have mentioned, you could use
sponge for this. But
sponge is not available to everyone.
The following is a generic solution that involve just the shell:
exec 3<file; rm file; rev <&3 >file; exec 3<&-
This opens the file (as fd 3) and deletes it. More precisely, this only deletes the directory entry for the file, not the file itself. The file won't be deleted until all hardlinks to it are deleted and all handles to it are closed.
rev is run reading from the "deleted file". It's output is sent to a new file. While this new file has the same name as the original file, it's a different file. As such, there's no conflict.
Finally, we close the descriptor to the original file, allowing it to be freed.
The problem with the above approach is that the data is lost if a problem occurs. That is why one might prefer the following (which uses no more disk space than the above):
( rev file >file.new && mv file.new file ) || rm file.new