Very often I do simple operations on a file, for example:

cat file1.txt|sed -r 's/^ *//'

to remove spaces at the beginning of the line. If I want to overwrite the file, the only way I know is:

cat file1.txt|sed -r 's/^ *//' > file2.txt
mv file2.txt file1.txt

This is very ennoying, because I have to check if file2.txt exists, because I have to write two commands instead of one, and so on.

So I thought: is there a way to do a full file buffering inside a chain of piped commands? So i could write:

cat file1.txt| magicbuffercommand |sed -r 's/^ *//' > file1.txt

This command should buffer (until a maximum of bytes, of course) and wait for an EOF, and then should start to write to stdout.

Is there anything that can do such a thing?

  • To all people saying things like "you can do it with -i switch of sed" or "useless use of cat command", and similar, this is just an example. Replace it with any other command, maybe it's sed, maybe it's "cat -n", maybe it's sort, and so on – d3k Dec 7 '14 at 11:55

You don't need the cat. sed happily accepts the file name as argument:

sed 's/^ *//' <file>

If you use GNU sed you can use the -i or --in-place switch to edit the file in place:

sed -i 's/^ *//' <file>

To answer the question, you can achieve “full file buffering” using the tool sponge from the moreutils package. Using sponge you can do:

<command> <file> | sponge <file>  # or
<command1> < <file> | <command2> | sponge <file>

Using your sed example this becomes:

sed 's/^ *//' <file> | sponge <file>
| improve this answer | |
  • Wonderful, sponge is exact what I was looking for! – d3k Dec 7 '14 at 11:58

Have you looked at the Linux buffer command? This utilizes a user shared memory segment to basically allow concurrent reads/writes. I suppose that it could buffer an entire file if the shared memory segment is large enough.

The buffer command might not be automatically installed, but I have found the program in many of the repositories on different Linux distributions. At worse, you can google and find the source and compile/link it yourself.

I have used buffer to speed up writes to slower devices such as tape drives and it does reduce the transfer time about 10-20%.

| improve this answer | |

One other possibility is to put the entire contents of a file into a shell variable. There used to be a limit on the size, but I understand that this is no longer a problem. As long as you have the memory (of course exceeding physical memory would cause swapping) you can try the following:

For example:

 varx=`cat filename`
 echo "$varx" | sed ..... >$filename

I only mention this in the event you may want to so something other than a sed command.

| improve this answer | |

You simply have to use -i switch of (GNU) , so

sed -i -r 's/^ *//' file_to_replace_in_place.txt 

and for Os X :

sed -i.'' -r 's/^ *//' file_to_replace_in_place.txt 

Another (more generic) solution is to use :

cat file | sed 's/^ *//' | tee file

you have to take care of huge files, this can exit without error, nor change.

| improve this answer | |

Useless use of the cat command. Use sed directly to print the contents or use -i to do an in place edit.

| improve this answer | |

You can use Vim in Ex mode:

ex -sc '%s/^ *//|x' file1.txt
  1. % select all lines

  2. s substitute

  3. x save and close

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.