Pipe result to multiple commands without writing on screen

Question

I have a very big zip file (~10 GB) which I want to unzip, and the result has to be passed to two commands. The results of both subsequent commands have to be appended to a file, and

Currently, I run

unzip -p bigFile.zip | head -n 1 >> output.txt
unzip -p bigFile.zip | grep -v 'skipLine' >> output.txt

which gives about a million rows. Now, I want to do it in 1 line. Using tee, I came up with

unzip -p bigFile.zip | tee >(head -n 1 >> output.txt) >(grep -v 'skipLine' 
>> output.txt)

But then the output.txt file only contains 51 rows, of which the last line is not even complete. Also, the latter command prints the results to the terminal, which I do not want.

I have also tried

unzip -p bigFile.zip | tee >(head -n 1 >> output.txt) | grep -v 'skipLine' 
>> output.txt

but unfortunately, this did not work either.

Any help is greatly appreciated! It is not necessary to use tee, any other command works fine, as long as I can pass the output of the unzip part to both the head and grep commands.

EDIT: It would be even better if the outputs of the head and grep commands could be 'combined' and then passed to zip. Is this possible?

Answer 1

You can unzip, and pipe into a shell script that consumes and prints the first line and allows grep to consume the rest:

unzip -p bigFile.zip | {
    IFS= read -r header       # consume the first line, verbatim
    printf "%s\n" "$header"
    grep -v 'skipLine' 
} >> output.txt

Everything printed from the block is appended to the output file.

Answer 2

I can't think of a way to pipe to two separate commands like that, but you can use awk instead:

unzip -p file.zip | awk 'NR==1 || !/skipLine/' > output

The awk command will print the current line if it is the first line (NR==1) or if it doesn't match skipline (!/skipLine).

The best I can come up with combining grep and head is:

unzip -p file.zip | tee >(head -n1 > output) | tail -n+2 | grep -v skipLine >> output

The idea is to first catch the first line with head -n1 and then use tail -n+2 to print everything except the first line and this is then passed to grep. But just use the awk approach, it's simpler and easier.

Answer 3

Perhaps I'm missing a subtle point, but it looks like you want to keep the first line and then anything after the first that does not contain the unique text skipLine. Seems like sed would handle that nicely.

#!/bin/bash
sed -n '1 {p;d}         #for line 1, print and discard
        /skipLine/d     #delete any line with skipLine
        p               #print remainder
' << End_of_SampleData
head
prefix skipLine
keepLine
skipLine suffix
keepLine
End_of_SampleData

Output:

head
keepLine
keepLine

You can just pipe your data into this script unzip -p bigFile.zip | sed ... if this test case matches your requirements. This should be pretty quick since there is just one unzip and one sed process to run.

Answer 4

There are two problems you're experiencing with your tee command:

tee >(head -n 1 >>output.txt) >(grep -v 'skipLine' >>output.txt)

The easy one is that you want to discard the output. To do that, you could simply send standard output to /dev/null:

tee >(head -n 1 >>output.txt) >(grep -v 'skipLine' >>output.txt) >/dev/null

But that seems wasteful; it's more efficient connect the second command to tee's standard out:

tee >(head -n 1 >>output.txt) | grep -v 'skipLine' >>output.txt

The second is that head closes its input after reading as much input as it needs. The simple fix is to follow that with a process that discards its input:

tee >(head -n 1 >>output.txt; dd of=/dev/null) | grep -v skipLine >>output.txt

We might replace the two opens of output.txt with a single one, if we want this pipeline to replace any previous content:

exec 3>output.txt
tee >(head -n 1 >&3; dd of=/dev/null) | grep -v skipLine >&3

Do note that in all these versions, there's no guarantee that the head line will be at the beginning of the output file, unlike the initial version with two separate pipelines. If that's a requirement, then look at one of the other answers for replacements (or take a gamble with an arbitrary sleep sequenced before the grep).

But I hope this answer helps you understand why you see the results that you do, rather than just spoon-feeding alternatives.