1

I have a very big zip file (~10 GB) which I want to unzip, and the result has to be passed to two commands. The results of both subsequent commands have to be appended to a file, and

Currently, I run

unzip -p bigFile.zip | head -n 1 >> output.txt
unzip -p bigFile.zip | grep -v 'skipLine' >> output.txt

which gives about a million rows. Now, I want to do it in 1 line. Using tee, I came up with

unzip -p bigFile.zip | tee >(head -n 1 >> output.txt) >(grep -v 'skipLine' 
>> output.txt)

But then the output.txt file only contains 51 rows, of which the last line is not even complete. Also, the latter command prints the results to the terminal, which I do not want.

I have also tried

unzip -p bigFile.zip | tee >(head -n 1 >> output.txt) | grep -v 'skipLine' 
>> output.txt

but unfortunately, this did not work either.

Any help is greatly appreciated! It is not necessary to use tee, any other command works fine, as long as I can pass the output of the unzip part to both the head and grep commands.

EDIT: It would be even better if the outputs of the head and grep commands could be 'combined' and then passed to zip. Is this possible?

4

I can't think of a way to pipe to two separate commands like that, but you can use awk instead:

unzip -p file.zip | awk 'NR==1 || !/skipLine/' > output

The awk command will print the current line if it is the first line (NR==1) or if it doesn't match skipline (!/skipLine).

The best I can come up with combining grep and head is:

unzip -p file.zip | tee >(head -n1 > output) | tail -n+2 | grep -v skipLine >> output

The idea is to first catch the first line with head -n1 and then use tail -n+2 to print everything except the first line and this is then passed to grep. But just use the awk approach, it's simpler and easier.

5
  • 1
    My first reaction was unzip | { IFS= read -r header; echo "$header"; grep -v skipLine; } > file but I like the awk a lot better. – glenn jackman Feb 23 at 17:14
  • 1
    @Quasímodo whoops, I had forgotten to append the output to output in the second one, thanks. – terdon Feb 23 at 17:20
  • @glennjackman ooh, I hadn't considered using read to read and discard the first line, that's neat! Worth its own answer, I think. – terdon Feb 23 at 17:20
  • @terdon Welcome! I'm still worried about concurrency though: Aren't Grep and Head in a race condition there? – Quasímodo Feb 23 at 17:23
  • @Quasímodo I... don't know, actually. I don't think it would be an issue though since head -n1 consumes the 1st line and tail -n+2 only prints the rest. I think it is safe(ish) to assume that head -n1 > output will always finish before ` tail -n+2 | grep ...` so I think there will never be a problem. I would be interested to know if I'm wrong though! – terdon Feb 23 at 17:27
4

You can unzip, and pipe into a shell script that consumes and prints the first line and allows grep to consume the rest:

unzip -p bigFile.zip | {
    IFS= read -r header       # consume the first line, verbatim
    printf "%s\n" "$header"
    grep -v 'skipLine' 
} >> output.txt

Everything printed from the block is appended to the output file.

2

Perhaps I'm missing a subtle point, but it looks like you want to keep the first line and then anything after the first that does not contain the unique text skipLine. Seems like sed would handle that nicely.

#!/bin/bash
sed -n '1 {p;d}         #for line 1, print and discard
        /skipLine/d     #delete any line with skipLine
        p               #print remainder
' << End_of_SampleData
head
prefix skipLine
keepLine
skipLine suffix
keepLine
End_of_SampleData

Output:

head
keepLine
keepLine

You can just pipe your data into this script unzip -p bigFile.zip | sed ... if this test case matches your requirements. This should be pretty quick since there is just one unzip and one sed process to run.

2
  • {p;d} is different to what's in the question, where any line that passes both filters will appear twice. It seems we just want p there. – Toby Speight Feb 28 at 15:18
  • Oh, and that //d;p is more simply written //!p, since they are the last commands in the program. That gives a simpler script: sed -n -e 1p -e '/skipLine/!p' – Toby Speight Feb 28 at 16:03
0

There are two problems you're experiencing with your tee command:

tee >(head -n 1 >>output.txt) >(grep -v 'skipLine' >>output.txt)

The easy one is that you want to discard the output. To do that, you could simply send standard output to /dev/null:

tee >(head -n 1 >>output.txt) >(grep -v 'skipLine' >>output.txt) >/dev/null

But that seems wasteful; it's more efficient connect the second command to tee's standard out:

tee >(head -n 1 >>output.txt) | grep -v 'skipLine' >>output.txt

The second is that head closes its input after reading as much input as it needs. The simple fix is to follow that with a process that discards its input:

tee >(head -n 1 >>output.txt; dd of=/dev/null) | grep -v skipLine >>output.txt

We might replace the two opens of output.txt with a single one, if we want this pipeline to replace any previous content:

exec 3>output.txt
tee >(head -n 1 >&3; dd of=/dev/null) | grep -v skipLine >&3

Do note that in all these versions, there's no guarantee that the head line will be at the beginning of the output file, unlike the initial version with two separate pipelines. If that's a requirement, then look at one of the other answers for replacements (or take a gamble with an arbitrary sleep sequenced before the grep).

But I hope this answer helps you understand why you see the results that you do, rather than just spoon-feeding alternatives.

1
  • You asked for the outputs to be recombined - that's what happens in the last version, and it's not hard to redirect to a pipe rather than file. – Toby Speight Feb 28 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.