Remove extra header lines from file, except for the first line

Question

I have a file that looks like this toy example. My actual file has 4 million lines, about 10 of which I need to delete.

ID  Data1  Data2
1    100    100
2    100    200
3    200    100
ID  Data1  Data2
4    100    100
ID  Data1  Data2
5    200    200

I want to delete the lines that look like the header, except for the first line.

Final file:

ID  Data1  Data2
1    100    100
2    100    200
3    200    100
4    100    100
5    200    200

How can I do this?

Answer 1

You can use

sed '2,${/ID/d;}'

This will delete lines with ID starting from line 2.

Answer 2

header=$(head -n 1 input)
(printf "%s\n" "$header";
 grep -vFxe "$header" input
) > output

1. grab the header line from the input file into a variable
2. print the header
3. process the file with grep to omit lines that match the header
4. capture the output from the above two steps into the output file

Answer 3

For those who do not like curly brackets

sed -e '1n' -e '/^ID/d'

• n means pass line No.1
• d delete all matched line(s) that start with ^ID

Answer 4

Here are a couple more choices that don't require you to know the first line in advance:

perl -ne 'print unless $_ eq $k; $k=$_ if $.==1; 

The -n flag tells perl to loop over its input file, saving each line as $_. The $k=$_ if $.==1; saves the first line ($. is the line number, so $.==1 will only be true for the 1st line) as $k. The print unless $k eq $_ prints the current line if it isn't the same as the one saved in $k.

Alternatively, the same thing in awk:

awk '$0!=x;(NR==1){x=$0}' file 

Here, we test whether the current line is the same as what is saved in the variable x. If the test $0!=x evaluates to true (if the current line $0 is not the same as x), the line will be printed because the default action for awk on true expressions is to print. The first line (NR==1) is saved as x. Since this is done after checking whether the current line matches x, this ensures that the first line will also be printed.

Answer 5

Here's a fun one. You can use sed directly to strip all copies of the first line out and leave everything else in place (including the first line itself).

sed '1{h;n;};G;/^\(.*\)\n\1$/d;s/\n.*$//' input

1{h;n;} puts the first line into the hold space, prints it, and reads in the next line—skipping the rest of the sed commands for the first line. (It also skips that first 1 test for the second line, but that doesn't matter as that test wouldn't have applied to the second line.)

G appends a newline followed by the contents of the hold space to the pattern space.

/^\(.*\)\n\1$/d deletes the contents of the pattern space (thus skipping to the next line) if the portion after the newline (i.e. what was appended from the hold space) exactly matches the portion before the newline. This is where lines that duplicate the header will get deleted.

s/\n.*$// deletes the portion of text that was added by the G command, so that what gets printed is just the line of text from the file.

However, since regex is expensive, a slightly faster approach would be to use the same condition (negated) and Print up to the newline if the portion after the newline (i.e. what was appended from the hold space) doesn't exactly match the portion before the newline and then unconditionally delete the pattern space:

sed '1{h;n;};G;/^\(.*\)\n\1$/!P;d' input

Output when given your input is:

ID  Data1  Data2
1    100    100
2    100    200
3    200    100
4    100    100
5    200    200

Answer 6

AWK is a quite decent tool for such purpose as well. Here's sample run of code:

$ awk 'NR == 1 {print} NR != 1 && $0!~/ID  Data1  Data2/' rmLines.txt | head -n 10                                
ID  Data1  Data2
1    100    100
     100    200
3    200    100
1    100    100
     100    200
3    200    100
1    100    100
     100    200
3    200    100

Break down:

• NR == 1 {print} tells us to print first line of text file
• NR != 1 && $0!~/ID Data1 Data2/ logical operator && tells AWK to print line that is not equal to 1 and doesn't contain ID Data1 Data2. Note the lack of {print} part; in awk if a test condition is evaluated to true,it is assumed for line to be printed.
• | head -n 10 is just a tiny addition to limit output to only first 10 lines. Not relevant to the AWK part itself, only used for demo purpose.

If you want that in a file, redirect the output of the command by appending > newFile.txt at the end of command, like so:

awk 'NR == 1 {print} NR != 1 && $0!~/ID  Data1  Data2/' rmLines.txt > newFile.txt

How does it hold up ? Pretty good actually:

$ time awk 'NR == 1 {print} NR != 1 && $0!~/ID  Data1  Data2/' rmLines.txt > /dev/null                            
    0m3.60s real     0m3.53s user     0m0.06s system

Side note

The generated sample file was done with for looping from one to million and printing first four lines of your file (so 4 lines times million equals 4 millions of lines ), which took 0.09 seconds, by the way.

awk 'BEGIN{ for(i=1;i<=1000000;i++) printf("ID  Data1  Data2\n1    100    100\n     100    200\n3    200    100\n");  }' > rmLines.txt

Answer 7

Awk, adapting to any header automatically:

awk '( FNR == 1) {header=$0;print $0;}
     ( FNR > 1) && ($0 != header) { print $0;}'  file1  file2 ....

ie, on the first line, get the header and print it, and subsequent line DIFFERENT from that header get printed.

FNR = Number of Records in the current File, so that you can have multiple files and it will do the same in each of them.

Answer 8

For the sake of completeness, Perl solution IMO slightly more elegant than @terdon gave:

perl -i -p -e 's/^ID.*$//s if $. > 1' file

Answer 9

Just to push back on the question a little bit ... it looks like maybe your input is itself the result of catting several TSV files together. If you can back up a step in your processing pipeline (if you own that or can talk to the people who do) you could use a header-aware tool to concatenate the data in the first place, and thereby remove the problem of having to remove extra header lines.

For example, using Miller:

$ cat f1.tsv
ID  Data1 Data2
1 100 100
2 100 200
3 200 100
$ cat f2.tsv
ID  Data1 Data2
4 100 100
$ cat f3.tsv
ID  Data1 Data2
5 200 200

$ cat f1.tsv f2.tsv  f3.tsv
ID  Data1 Data2
1 100 100
2 100 200
3 200 100
ID  Data1 Data2
4 100 100
ID  Data1 Data2
5 200 200

$ mlr --tsvlite cat f1.tsv f2.tsv  f3.tsv
ID  Data1 Data2
1 100 100
2 100 200
3 200 100
4 100 100
5 200 200

Answer 10

$ awk 'NR==1{h=$0; print} $0!=h' file
ID  Data1  Data2
1    100    100
2    100    200
3    200    100
4    100    100
5    200    200