Thanks to @Steeldriver for originally reproducing this. I wasn't able to myself (at least not at first), but having both you and Steeldriver confirm it lead me to dig a little deeper.
This appears to be a manifestation of the behavior that was reported in this Github issue. It was closed as being not specifically a WSL issue, but likely a problem in the legacy Windows Console Host.
Console Host is the "old" Windows terminal interface, going back at least two decades. I don't know that the issue was ever reported against it, but it would have been unlikely to have been fixed anyway.
Windows Console Host is in the process of being replaced by Windows Terminal, a (relatively) new, open-source terminal developed by Microsoft with modern terminal features. It can currently run as the default terminal on Windows 11. It can also run (just not as the default) on Windows 10.
I have confirmed that the problem does not occur on Windows Terminal.
Since the Github issue was from before WSL2 came out, that gave me a hunch as to why Steeldriver only saw it on one system, and I originally didn't see it. Interestingly, the problem (for me at least) only occurs in WSL1, not WSL2. This means that there's likely some incompatibility in the syscall translation interface in WSL1 that is triggering it. WSL2's Linux kernel does not have the issue.
So to resolve, either:
- Use Windows Terminal (or any non-Console Host terminal on Windows)
- Or use a WSL2 instance.
The former is my preference, since you can continue to use both WSL1 and WSL2 while taking advantage of the features of the upgraded Windows Terminal.
printf
to work as you expect (or to get a description of why it doesn't work), or do you want help with adding a BOM to the data stream (regardless of whether this is done withprintf
or not)?{ printf "\xFE\xFF"; cat some_utf-16_file.pag ; } | awk -f breakdown_on_invoice_nb.awk
. Get printf to work would be nice too. Understanding would be a bonus.LC_ALL=C
temporarily may sidestep the issue./usr/bin/printf
. Curiously it appears to work fine on another machine with (as far as I can tell) the same Windows build version and WSL locale.