1

I am trying to remove a line from a file using grep -v or sed but the problem is the number of line in the result is still the same after removal.

$ grep $'\t59116613\t' ../chr/19.TB0002183.all.vcf 
19      59116613        .       T       C       20.3144 .       DP=42;VDB=0.566336;SGB=-0.651104;RPB=0.504878;MQB=0.0279668;MQSB=0.943468;BQB=0.362091;MQ0F=0;ICB=1;HOB=0.5;AC=1;AN=2;DP4=21,13,2

$ grep -v $'\t59116613\t' ../chr/19.TB0002183.all.vcf | wc -l
108067

$ wc -l ../chr/19.TB0002183.all.vcf 
108067
  • 1
    try with this... awk '$2!=59116613' ../chr/19.TB0002183.all.vcf | wc -l – Kamaraj Apr 6 '17 at 4:31
  • same result awk '$2!=59116613' ../chr/19.TB0002183.all.vcf | wc -l 108067 – user1946989 Apr 6 '17 at 5:15
  • 1
    Please check if the last line of the file is terminated with a newline character. – VPfB Apr 6 '17 at 5:29
  • yes it seems to be a newline at the last line tail ../chr/13.TB0002183.all.vcf -n 1 | grep $'\n' 13 115109637 . C T 79 . DP=24;VDB=0.0742966;SGB=-0.636426;RPB=1;MQB=0.798183;MQSB=0.758709;BQB=0.293091;MQ0F=0;ICB=1;HOB=0.5;AC=1;AN=2;DP4=5,9,0,7;MQ=55 GT:PL 0/1:112,0,242 But how can it affect wc ? – user1946989 Apr 6 '17 at 5:49
  • @user1946989 wc -l count newlines. If the last line is not \n terminated, it is not counted. OTOH, grep always prints a newline after a line. That would explain the difference. A better check would be tail -1 FILENAME | od -c. – VPfB Apr 6 '17 at 6:07
1

When line counting gives unexpected results, we should check if each text processing utility understands what a line is the same way. This usually boils down to two questions:

1) File conversion issues: What charactes(s) are used as the EOL (end-of-line aka newline or line break)? It could be \r, \n, or \r\n depending on the OS.

2) Is the very last line also terminated with the EOL? If not, how the text processing utilities handle it?

In this case the second question was leading to an explanation.

What does the wc -l count?

Directly from the man page:

-l, --lines print the newline counts

Obviously, the last line is not counted, if it happens to be not terminated with a newline, i.e. the result is then one less than the real number of lines.

What does grep produce?

grep terminates each output line with a newline. I did not find it in the documentation, only tested it.


We can now make a test to explain the behaviour described in the question:

  1. Let's create a test file with 3 lines but without a trailing newline

echo -ne "first\nsecond\nthird" > 3lines.txt

  1. Let's check the last character:

tail -1 3lines.txt | od -c

Ok, no '\n'.

  1. wc -l < 3lines.txt

Output is: 2 (last line not counted, because last newline is missing)

  1. Pass all lines through grep

grep . < 3lines.txt | wc -l

Output is 3 (all lines counted, because no newline is missing)

  1. Filter out just one line (it can be any line)

grep -v first < 3lines.txt | wc -l

Output is 2 (again all lines counted).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.