3

Why does this pipeline truncate the file:

awk '...' file | tee file

while cating it first doesn't?

cat file | awk '...' | tee file

The file is just a regular config file with hundreds of lines. And I use tee to avoid using sponge (as I have to install moreutils first).

Improve this question
5
  • I would simplify the task head -1 file | tee -a file with utils cat, head, tail
    – nezabudka
    Sep 21, 2020 at 4:38
  • How big was the file? If the file is say 100k do you still find that cating the file avoids truncation? If you have more-utils package installed, why do you think someone wrote sponge? If the file is small, do you expect awk '...' file | { sleep 2 ; tee file} to work any better?
    – icarus
    Sep 21, 2020 at 5:35
  • @icarus It's a config file. I want to modify the config file across machines and it's not guaranteed that they have more-utils installed. So I use tee as awk cannot do inplace operation like sed's -i
    – annahri
    Sep 21, 2020 at 6:13
  • @nezabudka I think you missed the point. It's not about appending something, it's doing some operation in the file using awk then save it.
    – annahri
    Sep 21, 2020 at 6:15
  • @annahri, I see. I just gave an example without an unnecessary pipline, and the options you can omit at all
    – nezabudka
    Sep 21, 2020 at 18:37

1 Answer 1

Reset to default
10

You will find over the long run that this behavior is not consistent.

The problem is, that tee at some point in time opens the file for writing. From that moment on, the file is truncated. That is independent of which program reads the file.

It just happens that cat is very quick and that the reading of the file apparently has been finished before tee opens the file. But if for example the system load is bigger, cat may not have been finished, and the pipeline with cat may also truncate.

So, do not write to the same file as you are reading your pipeline from. Instead use Gnu awk's -i inplace, or if that is not available on your system, use

cp file file.tmp &&
awk '...' file.tmp | tee file
Improve this answer
7
  • 5
    Note that gawk -i does not work as gsed -i. The -i option to gawk takes an argument (a library to load). Using it to load the inline library will make edits happen in-line: gawk -i inline ....
    – Kusalananda ♦
    Sep 21, 2020 at 6:50
  • 4
    @Kusalananda it is inplace not inline as far as I know
    – Sundeep
    Sep 21, 2020 at 7:44
  • @Sundeep You are correct, it is inplace.
    – Kusalananda ♦
    Sep 21, 2020 at 8:39
  • 2
    If you don't add a && at the end of the cp line then you'll zap file if the cp failed. Note that this breaks awks ability to print the original file name and it leaves the tmp file lying around after you're done with it so I'd actually use awk '...' file | tee file.tmp && mv file.tmp file instead.
    – Ed Morton
    Sep 23, 2020 at 12:12
  • @EdMorton: if this is in a script, you are absolutely right. But then, why would you use tee like this?
    – Ljm Dullaart
    Sep 23, 2020 at 16:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.