Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Possible Duplicate:
How can I make iconv replace the input file with the converted output?

I frequently connect to amazon ec2 using their public DNS names
(ec2-12-34-56-78.compute-1.amazonaws.com) and because of this my known_hosts file gets overwhelmed with a lot of ec2 entries I will never use again.

I know I could probably use sed -i to edit in place but I wanted to use grep and so I did this:

grep -v ec2 ~/.ssh/known_hosts > ~/.ssh/known_hosts

That leaves known_hosts as an empty file. If I do:

grep -v ec2 ~/.ssh/known_hosts > ~/.ssh/tmp
mv ~/.ssh/tmp ~/.ssh/known_hosts

then things are fine, but I am confused why reading and writing to the same file leaves it blank, and if there is any way around this when using grep, cat, etc.

share|improve this question

marked as duplicate by Gilles, Michael Mrozek Apr 12 '12 at 2:11

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

Even if the program were doing the redirection itself, opening the file for read and then for write before actually reading from it would result in reading nothing (unless the language's standard library speculatively prebuffered, but that means you can't configure the buffer first). What were you expecting to happen? – geekosaur Apr 11 '12 at 19:33
Sounds good. Thanks. Why is this not part of your answer? – cwd Apr 11 '12 at 19:58
Because it's actually a related issue, not the original problem (the shell is doing the file create, so grep would not have a chance to do anything about it anyway). I'm just confused as to how it would be expected to work in the first place. – geekosaur Apr 11 '12 at 20:00
This is not the first time I've experienced this, so I don't "expect" to to work anymore, but initially I assumed that input flows from left to right and you operate on it on the left side and then would output it to the file on the right side. Guess part of the problem is that years ago I was a windows user and something similar would work in DOS. – cwd Apr 11 '12 at 20:34
@cwd Hence the value of having this one closed as a duplicate, with a highly-visible link to the other question. – Gilles Apr 11 '12 at 23:06
up vote 7 down vote accepted

There is a utility called sponge that is a part of the moreutils suite. It was made for this exact purpose.

grep -v ec2 ~/.ssh/known_hosts | sponge ~/.ssh/known_hosts
share|improve this answer
Why not just piping to cat? Like: ... | cat > ~/.ssh/known_hosts ? – hek2mgl Jul 3 '14 at 13:47
cat can write to its output at any time; if it writes past the point at whatever is on the other side of the pipe has read, then you will read in what you wrote instead of the original file contents. That is what makes sponge special. It buffers its input until stdin closes, then writes. – Shawn J. Goff Jul 3 '14 at 13:54
Good to know. Thanks! – hek2mgl Jul 3 '14 at 13:56
This might be interesting: If sponge isn't available, you could emulate it using sed: ... | sed ':a;N;ba' – hek2mgl Jul 4 '14 at 11:49
sponge is also part of sbase – Sebastian Oct 16 '14 at 13:23

Redirections are done by the shell, before the command runs. This means that the shell is told to truncate the file before grep gets a chance to read it. There is no way around this if you are using shell redirection.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.