Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a file that looks like this toy example. My actual file has 4 million lines, about 10 of which I need to delete.

ID  Data1  Data2
1    100    100
2    100    200
3    200    100
ID  Data1  Data2
4    100    100
ID  Data1  Data2
5    200    200

I want to delete the lines that look like the header, except for the first line.

Final file:

ID  Data1  Data2
1    100    100
2    100    200
3    200    100
4    100    100
5    200    200

How can I do this?

share|improve this question
up vote 21 down vote accepted
header=$(head -n 1 file)
(printf "%s\n" "$header";
 tail -n +2 file|grep -vFxe "$header"
) > newfile
  1. grab the header line into a variable
  2. print the header
  3. print everything except the first line, and omit lines that look like the header
share|improve this answer
2  
or perhaps { IFS= read -r head; printf '%s\n' "$head"; grep -vF "$head" ; } <file – iruvar Jan 26 at 19:34
2  
or { head -n 1; grep -vF 'ID' ; } <infile – don_crissti Jan 27 at 0:43
    
Both good additions. Thanks to don_crissti for indirectly pointing out that posix recently removed -1 syntax from head, in favor of -n 1. – Jeff Schaller Jan 27 at 0:53
3  
@JeffSchaller, recently as in 12 years ago. And head -1 has been obsoleted for decades before that. – Stéphane Chazelas Jan 27 at 13:09

You can use

sed '2,${/ID/d;}'

This will delete lines with ID starting from line 2.

share|improve this answer
3  
nice; or to be more specific with the pattern matching, sed '2,${/^ID Data1 Data2$/d;}' file (using the right number of spaces between the columns, of course) – Jeff Schaller Jan 26 at 19:08
    
Hm I thought you could omit the semicolon for only 1 command, but ok. – bkmoney Jan 26 at 19:09
    
Not w/ sane seds, no. – mikeserv Jan 26 at 19:35
    
aaaand -i for the in-place edit win. – user2066657 Jan 26 at 22:28
4  
Or sed '1!{/ID/d;}' – Stéphane Chazelas Jan 27 at 13:01

For those who do not like curly brackets

sed -e '1n' -e '/^ID/d'
  • n means pass line No.1
  • d delete all matched line(s) that start with ^ID
share|improve this answer
5  
This can also be shorten to sed '1n;/^ID/d' filename. just a suggestion – val0x00ff Jan 26 at 19:36
    
Note that this will also print lines like IDfoo which are not the same as the header (unlikely to make a difference in this case, but you never know). – terdon Jan 27 at 12:39

Here's a fun one. You can use sed directly to strip all copies of the first line out and leave everything else in place (including the first line itself).

sed '1{h;n;};G;/^\(.*\)\n\1$/d;s/\n.*$//' input

1{h;n;} puts the first line into the hold space, and skips the rest of the sed commands for the first line (thus printing it, which is the default).

G appends a newline followed by the contents of the hold space to the pattern space.

/^\(.*\)\n\1$/d deletes the contents of the pattern space (thus skipping to the next line) if the portion after the newline (i.e. what was appended from the hold space) exactly matches the portion before the newline. This is where lines that duplicate the header will get deleted.

s/\n.*$// deletes the portion of text that was added by the G command, so that what gets printed is just the line of text from the file.

Output when given your input is:

ID  Data1  Data2
1    100    100
2    100    200
3    200    100
4    100    100
5    200    200
share|improve this answer
    
Related: vi.stackexchange.com/q/6269/4676 – Wildcard Jan 28 at 20:11

AWK is a quite decent tool for such purpose as well. Here's sample run of code:

$ awk 'NR == 1 {print} NR != 1 && $0!~/ID  Data1  Data2/' rmLines.txt | head -n 10                                
ID  Data1  Data2
1    100    100
     100    200
3    200    100
1    100    100
     100    200
3    200    100
1    100    100
     100    200
3    200    100

Break down:

  • NR == 1 {print} tells us to print first line of text file
  • NR != 1 && $0!~/ID Data1 Data2/ logical operator && tells AWK to print line that is not equal to 1 and doesn't contain ID Data1 Data2. Note the lack of {print} part; in awk if a test condition is evaluated to true,it is assumed for line to be printed.
  • | head -n 10 is just a tiny addition to limit output to only first 10 lines. Not relevant to the AWK part itself, only used for demo purpose.

If you want that in a file, redirect the output of the command by appending > newFile.txt at the end of command, like so:

awk 'NR == 1 {print} NR != 1 && $0!~/ID  Data1  Data2/' rmLines.txt > newFile.txt

How does it hold up ? Pretty good actually:

$ time awk 'NR == 1 {print} NR != 1 && $0!~/ID  Data1  Data2/' rmLines.txt > /dev/null                            
    0m3.60s real     0m3.53s user     0m0.06s system

Side note

The generated sample file was done with for looping from one to million and printing first four lines of your file (so 4 lines times million equals 4 millions of lines ), which took 0.09 seconds, by the way.

awk 'BEGIN{ for(i=1;i<=1000000;i++) printf("ID  Data1  Data2\n1    100    100\n     100    200\n3    200    100\n");  }' > rmLines.txt
share|improve this answer
    
Note that this will also print lines like ID Data1 Data2 foo which are not the same as the header (unlikely to make a difference in this case, but you never know). – terdon Jan 27 at 12:39
    
@terdon yes, exactly right. OP however specified only one pattern they want to remove and his example appears to support that – Serg Jan 27 at 13:38

Here are a couple more choices that don't require you to know the first line in advance:

perl -ne 'print unless $_ eq $k; $k=$_ if $.==1; 

The -n flag tells perl to loop over its input file, saving each line as $_. The $k=$_ if $.==1; saves the first line ($. is the line number, so $.==1 will only be true for the 1st line) as $k. The print unless $k eq $_ prints the current line if it isn't the same as the one saved in $k.

Alternatively, the same thing in awk:

awk '$0!=x;(NR==1){x=$0}' file 

Here, we test whether the current line is the same as what is saved in the variable x. If the test $0!=x evaluates to true (if the current line $0 is not the same as x), the line will be printed because the default action for awk on true expressions is to print. The first line (NR==1) is saved as x. Since this is done after checking whether the current line matches x, this ensures that the first line will also be printed.

share|improve this answer
    
I like not having to know the first line idea since it makes it a generalized script for your toolbox. – Mark Stewart Jan 27 at 21:53
1  
that awk method creates an empty/false array entry per distinct line; for 4M lines if all different (not clear from Q) and fairly short (appears so) this is probably okay, but if there are much more or longer lines this could thrash or die. !($0 in a) tests without creating and avoids this, or awk can do the same logic as you have for perl: '$0!=x; NR==1{x=$0}' or if the header line can be empty 'NR==1{x=$0;print} $0!=x' – dave_thompson_085 Jan 28 at 8:57
1  
@dave_thompson_085 where is an array per line created? You mean !a[$0]? Why would that create an entry in a? – terdon Jan 28 at 10:27
1  
Because that's how awk works; see gnu.org/software/gawk/manual/html_node/… especially the "NOTE". – dave_thompson_085 Jan 29 at 2:53
1  
@dave_thompson_085 well I'll be damned! Thanks, I was not aware of that. Fixed now. – terdon Jan 29 at 9:57

Awk, adapting to any header automatically:

awk '( FNR == 1) {header=$0;print $0;}
     ( FNR > 1) && ($0 != header) { print $0;}'  file1  file2 ....

ie, on the first line, get the header and print it, and subsequent line DIFFERENT from that header get printed.

FNR = Number of Records in the current File, so that you can have multiple files and it will do the same in each of them.

share|improve this answer

For the sake of completeness, Perl solution IMO slightly more elegant than @terdon gave:

perl -i -p -e 's/^ID.*$//s if $. > 1' file
share|improve this answer
1  
Ah, but the whole point of mine was to avoid the need to specify the pattern and instead read it from the first line. Your approach will simply delete any line that starts with ID. You have no guarantee that this won't delete lines that should be kept. Since you brought up elegance, g is pointless if you use ^ and $. In fact, all of your options to m/// are useless here except s; they activate features you're not using. So's the $, s/^ID.*//s would do the same thing. – terdon Jan 28 at 10:37
    
@terdon, fair enough. Yours is much more universal! – KWubbufetowicz Jan 28 at 19:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.