How to count the number of lines in a UTF-16LE/CR-LF/BOM file?

Question

The immediate thought is wc, but then the next not-so-immediate thought is... Is *nix's wc purely for *nix line endings \x0a?... It seems so.

I've semi-wangled my way around it, but I feel there may/must be a simpler way than working on a hex-dump of the original.

Here is my version, but there is still a mysterious discrepancy in the tallies. wc reports 1 more 0a than the sum of this script's CRLF + 0a.

 file="nagaricb.nag"
 echo Report on CR and LF in UTF-16LE/CR-LF
 echo =====================================
 cat "$file" | # a useles comment, courtesy of cat 
   xxd -p -c 2 |
     sed -nr '
       /0a../{
           /0a00/!{
               i ‾‾`0a:   embedded in non-newline chars       
               b
           }
       }
       /0d../{
           /0d00/!{
               i ‾‾`0d:   embedded in non-newline chars       
               b
           }
       }
       /0a00/{
           i ‾‾`CR: found stray 0a00       
           b
        }
       /0d00/{
           N
           /0d00\n0a00/{
               i ‾‾`CRLF: found as normal newline pairs
               b
           }
           i ‾‾`LF: found stray 0d00
        }' |
         sort |
           uniq -c
 echo "  ====="
 printf '  %s ‾‾`wc\n' $(<"$file" wc -l)

Output

Report on CR and LF in UTF-16LE/CR-LF
=====================================
    125 ‾‾`0a:   embedded in non-newline chars       
    407 ‾‾`0d:   embedded in non-newline chars       
  31826 ‾‾`CRLF: found as normal newline pairs
  =====
  31952 ‾‾`wc

---

Is there some more standard/simple way to do this?

Answer 1

I would convert the file to UTF-8 with LF line endings, so I can directly use the native tools:

$ iconv -f UTF-16LE -t UTF-8 myfile.txt | dos2unix | wc -l

The dos2unix part is the trickiest bit. There are many variants of this tool floating around, not all of which know how to be used in a pipeline. Sometimes it's called something else, like d2u.

Answer 2

Here's a perl script that opens files (given as command line arguments) in UTF-16 (endianness detected via BOM), and counts the lines.

#! /usr/bin/env perl
use strict;
use warnings;

while (my $file = shift @ARGV) {
    my $fh;
    if (!open($fh, '<:encoding(UTF-16)', $file)) {
        print STDERR "Failed to open [$file]: $!\n";
        next;
    }
    my $count = 0;
    $count++ while (<$fh>);
    print "$file: $count\n";
    close $fh;
}

(Dies if the BOM is not understood.)

Answer 3

If you have dos2unix version >= 7.1 you can use the -i option to get information about the number of line breaks. UTF-16 files are also supported. When the file has a BOM dos2unix automatically detects it is UTF-16, LE or BE. When the file has no BOM you can use option -ul to tell it is UTF-16LE (or -ub for UTF-16BE).

dos2unix -i will print the number of DOS, Unix, and Mac line breaks in that order. Example (with BOM):

$ dos2unix -i utf16le.txt
      50       0       0  UTF-16LE  text    utf16le.txt

Without BOM:

$ dos2unix -ul -i utf16len.txt
      50       0       0  no_bom    text    utf16len.txt

See the manual for more information.