0

I have been using wc -l to check for the number of lines exist in my files. It worked fine always but not this time.

I have 120 big files that are supposed to have at least two lines in each of them. I have just done some text editing work on those files to remove and add new lines. I was trying to check the final number of line by using wc -l * as usual. The output showed that most of the files had only one line.

I opened up one of the file (which showed from the result of the command that it had only one line) with vim and I can see that it had exactly 2 lines. Exit vim and check again using wc -l, the number of line for that file then appeared as 2.

Does anyone have any idea with what happened over here? And how can I solve this problem instead of opening all 120 files with vim?

PS: The final line of my files weren't empty.

  • I don't see from your question what the error is exactly. – schaiba Nov 24 '19 at 6:58
  • @schaiba Sorry if I wasn't clear enough. The wc -l command doesn't show precise line number for my files here. But it was corrected only after I tried editing the file using vim – web Nov 24 '19 at 7:16
  • 1
    Does Vim show [noeol] at the bottom when you first open one of these files with the editor? – Kusalananda Nov 24 '19 at 8:36
  • 1
    @web So, the last line would then not be counted by wc -l since wc -l counts newlines. – Kusalananda Nov 24 '19 at 9:20
  • 1
    @web, you need to either alter your idea of what the wc program does (it doesn't count "lines" it counts newline characters) or you need to fix your files so they end with newline characters. I give a couple of ways of how to do that in my answer. – icarus Nov 24 '19 at 9:57
6

The common gnu implementation of wc says

‘wc’ counts the number of bytes, characters, whitespace-separated words,
and newlines in each given FILE, or standard input if none are given or for a FILE of ‘-’.

so if there is no final newline character in the file the "lines" part of the wc output will be one less than expected. For example the following will output 1

printf 'hello\nworld' | wc -l 

The OP has confirmed in comments that vim is reporting the lack of the final newline. A simple fix if all the files are known to have this problem is

 for f in *
 do
     echo >> "$f"
 done

to append a newline to each file.

A way to add a newline conditionally to the end of all the files if they are missing one is to use sed.

sed -s -i '$s/$/\n/;P;d' *

uses some gnu extensions, -s to treat each file separately, -i to do an in place edit, and allowing \n to represent a newline. The sed program itself says on the last line of each file append a newline, and for each line print up to the first newline and move onto the next line.

  • Thanks for the response. I might cause some confusion on the term "newlines" here. The new lines that I have added in or deleted contained character. Btw, regarding the sayings that "It is likely that something in this work removed the final newline.", I actually check it with vim after the edit and the total number of line was not precisely shown in wc output. – web Nov 24 '19 at 8:19
  • If you take a file that you have not fixed, and run hexdump -C on it, is the final character shown as 0a in the hex part of the display? – icarus Nov 24 '19 at 8:28
  • @web Removed comment the the editing might have removed the final newline. – icarus Nov 24 '19 at 9:27
  • not really sure what do you mean by final character in the hex part. Would you mind to further explain about it? – web Nov 24 '19 at 9:46
  • 1
    @web, od -c is also usefull for this kind of analisys. – JJoao Nov 24 '19 at 13:02
1

This is not exactly an answer, but sharing a micro personal tool I use often to normalize text files (txtnorm):

#!/usr/bin/perl -spi
our($s);
s/\n\r|\r\n|\n|\r/\n/g;                 ## normalize \n
s/^(\xFF\xFE|\xFE\xFF|\xEF\xBB\xBF)//;  ## remove BOM !
s/(?<=.)\z/\n/;                         ## ensure newline at eof

if($s){ s/\xC2\xA0/ /g }                ## -s non breaking spaces-> " "

txtnorm *.txt normalizes end-of-lines, ensures newlines at eof, removes BOMs and (with -s) may normalize non-breaking spaces.

Be sure to use it JUST on text files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.