Error in wc command to read number of lines in file

Question

I have been using wc -l to check for the number of lines exist in my files. It worked fine always but not this time.

I have 120 big files that are supposed to have at least two lines in each of them. I have just done some text editing work on those files to remove and add new lines. I was trying to check the final number of line by using wc -l * as usual. The output showed that most of the files had only one line.

I opened up one of the file (which showed from the result of the command that it had only one line) with vim and I can see that it had exactly 2 lines. Exit vim and check again using wc -l, the number of line for that file then appeared as 2.

Does anyone have any idea with what happened over here? And how can I solve this problem instead of opening all 120 files with vim?

PS: The final line of my files weren't empty.

Answer 1

The common gnu implementation of wc says

‘wc’ counts the number of bytes, characters, whitespace-separated words,
and newlines in each given FILE, or standard input if none are given or for a FILE of ‘-’.

so if there is no final newline character in the file the "lines" part of the wc output will be one less than expected. For example the following will output 1

printf 'hello\nworld' | wc -l 

The OP has confirmed in comments that vim is reporting the lack of the final newline. A simple fix if all the files are known to have this problem is

 for f in *
 do
     echo >> "$f"
 done

to append a newline to each file.

A way to add a newline conditionally to the end of all the files if they are missing one is to use sed.

sed -s -i '$s/$/\n/;P;d' *

uses some gnu extensions, -s to treat each file separately, -i to do an in place edit, and allowing \n to represent a newline. The sed program itself says on the last line of each file append a newline, and for each line print up to the first newline and move onto the next line.

Answer 2

This is not exactly an answer, but sharing a micro personal tool I use often to normalize text files (txtnorm):

#!/usr/bin/perl -spi
our($s);
s/\n\r|\r\n|\n|\r/\n/g;                 ## normalize \n
s/^(\xFF\xFE|\xFE\xFF|\xEF\xBB\xBF)//;  ## remove BOM !
s/(?<=.)\z/\n/;                         ## ensure newline at eof

if($s){ s/\xC2\xA0/ /g }                ## -s non breaking spaces-> " "

txtnorm *.txt normalizes end-of-lines, ensures newlines at eof, removes BOMs and (with -s) may normalize non-breaking spaces.

Be sure to use it JUST on text files.