I have a file that looks like this toy example. My actual file has 4 million lines, about 10 of which I need to delete.

ID  Data1  Data2
1    100    100
2    100    200
3    200    100
ID  Data1  Data2
4    100    100
ID  Data1  Data2
5    200    200

I want to delete the lines that look like the header, except for the first line.

Final file:

ID  Data1  Data2
1    100    100
2    100    200
3    200    100
4    100    100
5    200    200

How can I do this?

up vote 26 down vote accepted
header=$(head -n 1 input)
(printf "%s\n" "$header";
 grep -vFxe "$header" input
) > output
  1. grab the header line from the input file into a variable
  2. print the header
  3. process the file with grep to omit lines that match the header
  4. capture the output from the above two steps into the output file
  • 2
    or perhaps { IFS= read -r head; printf '%s\n' "$head"; grep -vF "$head" ; } <file – iruvar Jan 26 '16 at 19:34
  • Both good additions. Thanks to don_crissti for indirectly pointing out that posix recently removed -1 syntax from head, in favor of -n 1. – Jeff Schaller Jan 27 '16 at 0:53
  • 3
    @JeffSchaller, recently as in 12 years ago. And head -1 has been obsoleted for decades before that. – Stéphane Chazelas Jan 27 '16 at 13:09

You can use

sed '2,${/ID/d;}'

This will delete lines with ID starting from line 2.

  • 3
    nice; or to be more specific with the pattern matching, sed '2,${/^ID Data1 Data2$/d;}' file (using the right number of spaces between the columns, of course) – Jeff Schaller Jan 26 '16 at 19:08
  • Hm I thought you could omit the semicolon for only 1 command, but ok. – bkmoney Jan 26 '16 at 19:09
  • Not w/ sane seds, no. – mikeserv Jan 26 '16 at 19:35
  • aaaand -i for the in-place edit win. – user2066657 Jan 26 '16 at 22:28
  • 4
    Or sed '1!{/ID/d;}' – Stéphane Chazelas Jan 27 '16 at 13:01

For those who do not like curly brackets

sed -e '1n' -e '/^ID/d'
  • n means pass line No.1
  • d delete all matched line(s) that start with ^ID
  • 5
    This can also be shorten to sed '1n;/^ID/d' filename. just a suggestion – Valentin Bajrami Jan 26 '16 at 19:36
  • Note that this will also print lines like IDfoo which are not the same as the header (unlikely to make a difference in this case, but you never know). – terdon Jan 27 '16 at 12:39

Here's a fun one. You can use sed directly to strip all copies of the first line out and leave everything else in place (including the first line itself).

sed '1{h;n;};G;/^\(.*\)\n\1$/d;s/\n.*$//' input

1{h;n;} puts the first line into the hold space, prints it, and reads in the next line—skipping the rest of the sed commands for the first line. (It also skips that first 1 test for the second line, but that doesn't matter as that test wouldn't have applied to the second line.)

G appends a newline followed by the contents of the hold space to the pattern space.

/^\(.*\)\n\1$/d deletes the contents of the pattern space (thus skipping to the next line) if the portion after the newline (i.e. what was appended from the hold space) exactly matches the portion before the newline. This is where lines that duplicate the header will get deleted.

s/\n.*$// deletes the portion of text that was added by the G command, so that what gets printed is just the line of text from the file.

However, since regex is expensive, a slightly faster approach would be to use the same condition (negated) and Print up to the newline if the portion after the newline (i.e. what was appended from the hold space) doesn't exactly match the portion before the newline and then unconditionally delete the pattern space:

sed '1{h;n;};G;/^\(.*\)\n\1$/!P;d' input

Output when given your input is:

ID  Data1  Data2
1    100    100
2    100    200
3    200    100
4    100    100
5    200    200

Here are a couple more choices that don't require you to know the first line in advance:

perl -ne 'print unless $_ eq $k; $k=$_ if $.==1; 

The -n flag tells perl to loop over its input file, saving each line as $_. The $k=$_ if $.==1; saves the first line ($. is the line number, so $.==1 will only be true for the 1st line) as $k. The print unless $k eq $_ prints the current line if it isn't the same as the one saved in $k.

Alternatively, the same thing in awk:

awk '$0!=x;(NR==1){x=$0}' file 

Here, we test whether the current line is the same as what is saved in the variable x. If the test $0!=x evaluates to true (if the current line $0 is not the same as x), the line will be printed because the default action for awk on true expressions is to print. The first line (NR==1) is saved as x. Since this is done after checking whether the current line matches x, this ensures that the first line will also be printed.

  • I like not having to know the first line idea since it makes it a generalized script for your toolbox. – Mark Stewart Jan 27 '16 at 21:53
  • 1
    that awk method creates an empty/false array entry per distinct line; for 4M lines if all different (not clear from Q) and fairly short (appears so) this is probably okay, but if there are much more or longer lines this could thrash or die. !($0 in a) tests without creating and avoids this, or awk can do the same logic as you have for perl: '$0!=x; NR==1{x=$0}' or if the header line can be empty 'NR==1{x=$0;print} $0!=x' – dave_thompson_085 Jan 28 '16 at 8:57
  • 1
    @dave_thompson_085 where is an array per line created? You mean !a[$0]? Why would that create an entry in a? – terdon Jan 28 '16 at 10:27
  • 1
    Because that's how awk works; see gnu.org/software/gawk/manual/html_node/… especially the "NOTE". – dave_thompson_085 Jan 29 '16 at 2:53
  • 1
    @dave_thompson_085 well I'll be damned! Thanks, I was not aware of that. Fixed now. – terdon Jan 29 '16 at 9:57

AWK is a quite decent tool for such purpose as well. Here's sample run of code:

$ awk 'NR == 1 {print} NR != 1 && $0!~/ID  Data1  Data2/' rmLines.txt | head -n 10                                
ID  Data1  Data2
1    100    100
     100    200
3    200    100
1    100    100
     100    200
3    200    100
1    100    100
     100    200
3    200    100

Break down:

  • NR == 1 {print} tells us to print first line of text file
  • NR != 1 && $0!~/ID Data1 Data2/ logical operator && tells AWK to print line that is not equal to 1 and doesn't contain ID Data1 Data2. Note the lack of {print} part; in awk if a test condition is evaluated to true,it is assumed for line to be printed.
  • | head -n 10 is just a tiny addition to limit output to only first 10 lines. Not relevant to the AWK part itself, only used for demo purpose.

If you want that in a file, redirect the output of the command by appending > newFile.txt at the end of command, like so:

awk 'NR == 1 {print} NR != 1 && $0!~/ID  Data1  Data2/' rmLines.txt > newFile.txt

How does it hold up ? Pretty good actually:

$ time awk 'NR == 1 {print} NR != 1 && $0!~/ID  Data1  Data2/' rmLines.txt > /dev/null                            
    0m3.60s real     0m3.53s user     0m0.06s system

Side note

The generated sample file was done with for looping from one to million and printing first four lines of your file (so 4 lines times million equals 4 millions of lines ), which took 0.09 seconds, by the way.

awk 'BEGIN{ for(i=1;i<=1000000;i++) printf("ID  Data1  Data2\n1    100    100\n     100    200\n3    200    100\n");  }' > rmLines.txt
  • Note that this will also print lines like ID Data1 Data2 foo which are not the same as the header (unlikely to make a difference in this case, but you never know). – terdon Jan 27 '16 at 12:39
  • @terdon yes, exactly right. OP however specified only one pattern they want to remove and his example appears to support that – Sergiy Kolodyazhnyy Jan 27 '16 at 13:38

Awk, adapting to any header automatically:

awk '( FNR == 1) {header=$0;print $0;}
     ( FNR > 1) && ($0 != header) { print $0;}'  file1  file2 ....

ie, on the first line, get the header and print it, and subsequent line DIFFERENT from that header get printed.

FNR = Number of Records in the current File, so that you can have multiple files and it will do the same in each of them.

For the sake of completeness, Perl solution IMO slightly more elegant than @terdon gave:

perl -i -p -e 's/^ID.*$//s if $. > 1' file
  • 1
    Ah, but the whole point of mine was to avoid the need to specify the pattern and instead read it from the first line. Your approach will simply delete any line that starts with ID. You have no guarantee that this won't delete lines that should be kept. Since you brought up elegance, g is pointless if you use ^ and $. In fact, all of your options to m/// are useless here except s; they activate features you're not using. So's the $, s/^ID.*//s would do the same thing. – terdon Jan 28 '16 at 10:37
  • @terdon, fair enough. Yours is much more universal! – KWubbufetowicz Jan 28 '16 at 19:39

Just to push back on the question a little bit ... it looks like maybe your input is itself the result of catting several TSV files together. If you can back up a step in your processing pipeline (if you own that or can talk to the people who do) you could use a header-aware tool to concatenate the data in the first place, and thereby remove the problem of having to remove extra header lines.

For example, using Miller:

$ cat f1.tsv
ID  Data1 Data2
1 100 100
2 100 200
3 200 100
$ cat f2.tsv
ID  Data1 Data2
4 100 100
$ cat f3.tsv
ID  Data1 Data2
5 200 200

$ cat f1.tsv f2.tsv  f3.tsv
ID  Data1 Data2
1 100 100
2 100 200
3 200 100
ID  Data1 Data2
4 100 100
ID  Data1 Data2
5 200 200

$ mlr --tsvlite cat f1.tsv f2.tsv  f3.tsv
ID  Data1 Data2
1 100 100
2 100 200
3 200 100
4 100 100
5 200 200
  • 1
    Thank you for adding this tidbit. This will be extremely useful in the future, as most of my pipelines require joining and merging files from individual samples. – Gaius Augustus Nov 16 '16 at 21:16

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.