Warning regarding ‘>’
Unix beginners who have just learned about I/O redirection (<
and >
)
often try things like
command … input_file > the_same_file
or
command … < file > the_same_file
or, almost equivalently,
cat file | command … > the_same_file
(grep
, sed
, cut
, sort
, and spell
are examples of commands
that people are tempted to use in constructs like these.)
Users are surprised to discover that these scenarios result in the file becoming empty.
A nuance that doesn’t seem to be mentioned in the other answer can be found lurking
in the first sentence of the Redirection section of bash(1):
Before a command is executed, its input and output may be redirected
using a special notation interpreted by the shell.
The first five words should be bold, italic, underlined, enlarged, blinking,
colored red, and marked with a
icon,
to emphasize the fact that the shell performs the requested redirection(s)
before the command is executed.
And remember also
Redirection of output causes the file … to be opened for writing ….
If the file does not exist it is created; if it does exist it is truncated to zero size.
So, in this example:
sort roster > roster
the shell opens the roster
file for writing, truncating it
(i.e., discarding all its contents), before the sort
program starts running.
Naturally, nothing can be done to recover the data.
One might naïvely expect that
tr "[:upper:]" "[:lower:]" < poem > poem
might be better.
Because the shell handles redirections from left to right,
it opens poem
for reading (for tr
’s standard input)
before it opens it for writing (for standard output).
But it doesn’t help.
Even though this sequence of operations yields two file handles,
they both point to the same file.
When the shell opens the file for reading, the contents are still there,
but they still get clobbered before the program is executed.
So, what to do about it?
Solutions include:
Check whether the program you’re running
has its own, internal, capability to specify where the output goes.
This is often indicated by a -o
(or --output=
) token.
In particular,
sort -o roster roster
is roughly equivalent to
sort roster > roster
except, in the first case, the sort
program opens the output file.
And it’s smart enough not to open the output file
until after it has read all of the input file(s).
Similarly, at least some versions of sed
have a -i
(edit in place) option
that can be used to write the output back out to the input file
(again, after all the input have been read).
Editors like ed
/ex
, emacs
, pico
, and vi
/vim
allow the user to edit a text file and save the edited text in the original file.
Note that ed
(at least) can be used non-interactively.
vi
has a related feature.
If you type :%!command
Enter,
it will write the contents of the edit buffer out to command
, read the output,
and insert it into the buffer (replacing the original contents).
Simple but effective:
command … input_file > temp_file && mv temp_file input_file
This has the drawback that, if input_file
is a link,
it will (probably) be replaced by a separate file.
Also, the new file will be owned by you, with default protections.
In particular, this carries the risk that the file will be end up being world-readable,
even if the original input_file
wasn’t.
Variations:
command … input_file > temp_file && cp temp_file input_file && rm temp_file
which will still (potentially) leave the temp_file
world-readable.
Even better:
cp input_file temp_file && command … temp_file > input_file && rm temp_file
These preserve the link status, owner, and mode (protection) of the file,
potentially at the cost of twice as much I/O.
(You may need to use an option like -a
or -p
on cp
to tell it to preserve attributes.)
command … input_file > temp_file &&
cp --attributes-only --preserve=all input_file temp_file &&
mv temp_file input_file
(broken into separate lines only for readability)
This preserves the mode of the file (and, if you’re root, the owner),
but makes it owned by you (if you’re not root), and makes it a new, separate file.
This blog
(“In-place” editing of files) suggests and explains
{ rm input_file && command … > input_file; } < input_file
This requires that the command
be able to process standard input
(but almost all filters can).
The blog itself calls this a risky kludge and discourages its use.
And this will also create a new, separate file (not linked to anything),
owned by you and with default permissions.
The moreutils package has a command called sponge
:
command … input_file | sponge the_same_file
See this answer for more information.
Here’s something that came as a complete surprise to me:
syntaxerror says:
[Most of these solutions] will fail on a read-only file system,
where “read-only” means that your $HOME
will be writable,
but /tmp
will be read-only (by default).
For instance, if you have Ubuntu, and you’ve booted into the Recovery Console,
this is commonly the case.
Also, the here-document operator <<<
will not work there either,
as it requires /tmp
to be read/write
because it will write a temporary file into there as well.
(cf. this question includes an strace
’d output)
The following may work in that case:
So, what was the question?
This has been a popular topic on U&L; it is addressed in the following questions:
… and that’s not counting Super User or Ask Ubuntu.
I have incorporated a lot of the information from the answers to the above questions
here in this answer, but not all.
(I.e., for more information, read the above-listed questions and their answers.)
P.S. I have no affiliation with the blog that I cited, above.