0

Trying to use grep today, I ran into the familiar problem of the Byte Order Mark (BOM) in a Unicode file (UTF-8, in this case). Specifically, I was trying to find a file beginning with XYZ with the pattern grep '^XYZ', but of course grep treated the BOM as three separate characters and did not match the first line of the file if the first line started with XYZ. I even tried to update the regular expression to ignore spaces ('^[[:space:]]*XYZ'), but to no avail.

Other questions have dealt with converting files or targeting the BOM specifically, but I want to know if POSIX tools have a general option to handle Unicode files correctly. If grep handled the Unicode file correctly, it would consider the file contents to start after the BOM and match XYZ on the first line just like any other line.

  • 2
    POSIX doesn't have a concept of bytes in text files that aren't part of characters. grep "^\($(printf '\xef\xbb\xbf')\)\?XYZ" would work, modulo other zero-width no-break spaces starting later lines. – Michael Homer Jan 3 '18 at 1:08
  • ... and so I don't think there's an option or even a compliant locale that could do it, but I'm less certain about locales. Arguably this behaviour is correct, though unhelpful for your use case. – Michael Homer Jan 3 '18 at 1:11
3

The Unicode Consortium has an FAQ that includes How should I deal with BOMs. This portion includes:

Where a text data stream is known to be plain Unicode text (but not which endian), then BOM can be used as a signature. If there is no BOM, the text should be interpreted as big-endian.

and

Where the precise type of the data stream is known (e.g. Unicode big-endian or Unicode little-endian), the BOM should not be used. In particular, whenever a data stream is declared to be UTF-16BE, UTF-16LE, UTF-32BE or UTF-32LE a BOM must not be used.

Note that UTF-8 is always of known endianness, because it has no endianness. So as long as you know the text is UTF-8, "the BOM should not be used."

Even cat will return incorrect results when using a BOM unnecessarily, as the BOMs of all files but the first will be treated as zero-width non-breaking spaces. But, the power of UNIX lies in filters.

For operations on a single file or stream,sed "1s/^$(printf '\357\273\277')//" in a pipeline will strip a BOM if present, leaving all other streams intact.

For operations with multiple files, a shell with process substitution (like Bash, but unfortunately not POSIX shell) is useful:

sb() { sed "1s/$(printf '\357\273\277')//" "$@" ; }
cat <(sb file1) <(sb file2) …
  • Interesting; how, then, does any random text file mark itself as UTF-8 or ASCII without a BOM? – palswim Jan 5 '18 at 20:43
  • 1
    Note that this answer assumes you know the encoding. Plain ASCII is identical to UTF-8. "Extended ASCII" (aka arbitrary 8-bit character sets) frequently use sequences that would be invalid in UTF-8, so the distinction isn't difficult – Fox Jan 5 '18 at 21:56
  • Ah, right, the whole using 7 out of the 8 bits deal. – palswim Jan 10 '18 at 20:03
  • I think you misunderstood the meaning of the quoted text. It refers to out of band signalling: when you already know the stream encoding, there's no need to add an inband encoding marker. The part I'd have quoted from your source is this: "Where a text data stream is known to be plain text, but of unknown encoding, BOM can be used as a signature. If there is no BOM, the encoding could be anything." It's possible usually to guess the right one, but the purpose of the BOM is exactly to avoid this guesswork. The real issue is that POSIX tools have no concept of unicode semantics. – Matyas Koszik Jan 6 at 22:34
0

From the other answer, it sounds like I was dealing with files with an improper BOM signature.

So, the answer is that POSIX tools handle Unicode (UTF-8) files correctly already.

If you have bad Unicode, of course they don't handle it correctly, but you can use the BOM targeting from other questions to deal with superfluous BOM signatures.

0

Most POSIX tools operate on bytes, and not characters. Unicode signalling is meaningless to them, so it'll be treated like any other piece of data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.