2

Suppose I'm given a variety of files, some with line endings of \r, \n and \r\n. How would I efficiently count the number of lines in a file when I don't know what to expect beforehand?

Obviously wc -l won't work if there are Mac line endings.

Grep appears to be very inaccurate when used:

grep -o '\r' /path/to/file.txt | wc -l

Gives 1041 lines but there are actually only 299 lines, and nano is able to confirm that by auto-converting from Mac format.

It also doesn't provide the versatility I was hoping for because grep -o '[\r\n]+' /path/to/file.txt | wc -l returns a count of 0.

How do I do this correctly?

  • convert the files to end with \n and then use wc? – Sundeep Dec 29 '16 at 6:15
  • 1
    What do you want the number to be for a file like 1\n2\n3\r4? Is this 0, 1, 2, 3 or 4 lines? It doesn't end with a "line ending", so could be rejected as binary and have 0 lines, it could be a Mac file with 1 '\r' in it, it could be unix with 2 '\n' in it, it could be unix with 2 and a bit lines in it, or something with 3 and a bit lines in it. My point is that the concept of lines is a bit vague. What are you going to use the information for? – icarus Dec 29 '16 at 12:30
  • These are delimited files I'm scanning, usually tab or comma delimited. Each line is supposed to be terminated with a new line character, but that character will vary depending on the source. The files can be huge, so rewriting the entire file to use a different line ending would not be practical. I still need the line count. – eComEvo Dec 29 '16 at 14:13
  • Could you try cat file.txt | tr -c -d '\r' | wc -c to count the number of \r in the file? When I tried your grep command, I only found matches for the letter "r". – JigglyNaga Dec 30 '16 at 12:20
  • @JigglyNaga that returned the correct number of lines, but how memory efficient is piping it thru three commands like that? I know wc -l is crazy fast and memory efficient on large files. – eComEvo Dec 30 '16 at 15:45
0

Your grep command was giving far too many matches because grep uses Basic Regular Expressions, which don't give special meaning to \r -- it was counting occurrences of the letter "r".

tr does recognise \n and \r, so it can find all the matches, and let wc -c count them byte-by-byte:

tr -d -c '\n\r' < file.txt | wc -c

But as your file may also contain Windows/DOS line-endings (\r\n), those would get counted twice. You will need an additional step to convert any DOS-style line-endings to one of the other forms, then count individual line-end characters as before:

sed 's/\r$//' file.txt | tr -d -c '\n\r' | wc -c

Unfortunately, sed doesn't always recognize \r, either. You may need to use some other way to specify a carriage-return, such as the shell's own escaping, the echo command, or putting the sed script in a file.

  • Really only concerned with Mac, Unix and Windows files. DOS just doesn't happen for this use case so the first command works. Thanks! – eComEvo Jan 2 '17 at 23:55
  • Sorry, I was using "DOS line-endings" as a shorthand for "DOS and Windows", ie. \r\n, sometimes described as "CR LF". Edited for consistency. – JigglyNaga Jan 3 '17 at 10:17
0
gawk 'END {print NR}' RS='\r|\n|\r\n' file.txt

The RS variable specifies what is considered a newline to gawk, in this case a regular expression.

  • I would change RS='\r\n?|\n' but that's just stylistic. – phk Dec 29 '16 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.