Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

If I run:

cat messages.txt | sed -e 's/a/a/g' > messages.txt

on one large file (2500+ lines) I find that the resulting file will only have about 900 lines after the command in cygwin and will have no lines in gentoo. However if I run

cat messages.txt | sed -e 's/a/a/g' > other_messages.txt

it retains all the lines as it should.

My question is why and is there any way to do it other than

cat messages.txt | sed -e 's/a/a/g' > other_messages.txt
rm messages.txt
mv other_messages.txt messages.txt
share|improve this question
1  
For the curious, the reason the former doesn't work is sed reads files a block at a time. It reads in the first block, processes it, and writes it to standard out. Before it can read the next block, your shell takes the first block of output from sed and inserts it into the output file -- if that's the same as the input file, the shell has just overwritten it. When sed goes to read the next block, it no longer exists, so it stops –  Michael Mrozek Oct 5 '10 at 15:29
    
It seems that Michael was writing his comment when I was writing my additional answer. So many fast typers. –  Steven D Oct 5 '10 at 15:48
1  
@Michael: Actually I believe the output file gets truncated before anything else happens. –  Dennis Williamson Oct 5 '10 at 20:54
    
@Dennis When it gets opened it probably does, but I think the shell waits to open it; otherwise you'd never get any input in sed, and I've seen the behavior the OP describes where part of the file is there and then it cuts off –  Michael Mrozek Oct 5 '10 at 21:08

3 Answers 3

up vote 10 down vote accepted

Why don't you just write

sed -i -e 's/a/a/g' messages.txt

the -i means "in place"

share|improve this answer
    
Yes, that works, thanks! :D –  yuumei Oct 5 '10 at 12:12
1  
Warning: not all variants of sed implement this option. –  mouviciel Oct 6 '10 at 13:50

fschmitt's answer is the best when using sed; however, in a more general sense this anti-pattern:

cat infile | filter > infile

is likely to cause you a good number of problems. For instance if I have a file called infile that looks like this:

Hello
World

and run this command:

cat infile | tr "[:upper:]" "[:lower:]"

I get

hello
world

But if I run cat infile | tr "[:upper:]" "[:lower:]" > infile I will get an empty file. Why?

Well, when you use the output redirection operator > you are saying "Put my standard output into this file, and if that file exists overwrite it." Now you may think that this should work since your filter will return all the lines of the original file. However, what often ends up happening is that the shell will clobber your file before any lines are read. Then, your filter command will go to read lines from an empty file, find none, and thus return none. In some places you might get "lucky" enough to have some lines read before the file gets clobber, but it is best to just avoid this pattern altogether.

To get around this particular issue you have a few options. One is to simply do something like:

cat infile | filter > tmpfile; mv tmpfile infile

If you need to be sure that your temp file won't clobber some other file or have other nasty things happen to it, you should look into mktemp. (see man mktemp and info coreutils mktemp)

Another option is to use sponge from moreutils.

Also, many of these examples are examples of useless uses of cat.

share|improve this answer

Yet another (portable) way to edit a file in-place is to use ed.

# cf. http://wiki.bash-hackers.org/howto/edit-ed
cat <<-'EOF' | ed -s messages.txt
H
,g/a/s//b/g
wq
EOF


# ... or read the file contents into a variable, modify it and write it back to file
file_contents="$(cat messages.txt)"
printf '%s' "$file_contents" | sed -e 's/a/b/g' > messages.txt


# ... and, if you want to play around with a file descriptor hack, ...
# (As long as there's a fd associated with a file, the file can be accessed via the fd.) 

exec 3<messages.txt  # open file on fd 3 for reading
rm -f messages.txt
sed -e 's/a/b/g' <&3 > messages.txt
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.