So I went to the source, and it looks like the slowness is in handling double byte characters. Essentially, for every character read in, it needs to call
mbrtowc() to try to convert it to a wide character, then that wide character is tested to see if it's a word separator, line separator, etc.
Indeed, if I change my locale
LANG variable from the default
en_US.UTF-8 (UTF-8 is a multibyte character set) and set it to "
C" (simple single byte character set),
wc is able to use single-byte optimizations, which speeds it up considerably, taking only about a quarter as long as before.
Additionally, it only has to check each character if it's doing word (
-w), line length (
-L) or character (
-m) counts. If it's only doing byte and/or line counts, it can skip the wide character handling and then it runs extremely quickly -- faster than
I ran it through
gprof, and the functions that are used to handle the multibyte characters (
myiswprint(), etc) are taking up about 30% of the execution time alone, and the code that steps through the buffer is much more complex because it has to handle variable sized steps through the buffer for variable sized characters, as well as stuffing any partially completed characters that span the buffer back to the beginning of the buffer so it can be handled the next time around.
Now that I know what to look for, I found a few posts mentioning the utf-8 slowness with some utilities: