11

As a C programmer, I was surprised to see that wc -c (which count the number of bytes), and wc -m (which counts the number of characters) output very different results for a long, text file of mine. I had always been told that sizeof(char) is 1 byte.

qdii@nomada ~/Documents $ wc -c sentences.csv
102990983 sentences.csv
qdii@nomada ~/Documents $ wc -m sentences.csv
89023123 sentences.csv

Any explanation?

  • See @rici's answer below... you've got your -m and -c flags backwards in your question (c = bytes, m = characters)... your example output is correct, though. – Dan Oct 16 '12 at 19:16
19

The char type in C is one byte, but it's intended for ASCII characters; there are variable-width encodings like UTF-8 that can take up many bytes per character. wc uses the mbrtowc(3) function to decode multibyte sequences, depending on the locale set by the LC_CTYPE environment variable. If you set the locale properly, you should get the same result for all cases. For example:

qdii@nomada ~/Documents $ LC_CTYPE="C" wc -m sentences.csv
102990983 sentences.csv
15

At a guess,

  1. Your locale uses UTF-8 encoding, and

  2. About 10% of your file consists of characters which require more than one octet to encode into UTF-8.

By the way, from man wc:

   -c, --bytes
          print the byte counts

   -m, --chars
          print the character counts

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.