6

How can I check in bash whether a variable contains a valid UTF-8 string without any special control characters (such as newline or backspace or carriage return etc.)?

12

Provided the locale uses UTF-8 as its character encoding (check with locale charmap):

[[ $string =~ ^[^[:cntrl:]]*$ ]]

should work except that at least on GNU systems, it fails to reject strings that contain the UTF-8 encoding of characters with code points above 0x110000 (which are no longer valid in the current definition of UTF-8 encoding). That's down to the system's regexp library bash uses to do pattern matching.

That's the same for GNU expr and:

expr " $string" : ' [^[:cntrl:]]*$' > /dev/null

Here, you may want to switch to zsh where, in UTF-8 locales,

set -o extendedglob
[[ $string = [' '-$'\ud7fff'$'\ue000'-$'\U10FFFF']# && $string != *[[:cntrl:]]* ]]

should work consistently regardless of the system (at least for the matching of valid characters; you might find variations between systems for which character is classified as [:cntrl:]).

In zsh glob patterns, character ranges in multibyte character locales are based on the the wide character value of the character, which in UTF-8 is invariably the Unicode code point. Bytes that don't form part of valid characters are assigned wchar_t values in the 0xDC80..0xDCFF range in the second part of the 0xD800..0xDFFF UTF16 surrogate pairs which are not characters (a common approach sanctioned by Unicode to be able to handle arbitrary sequences of bytes).

Character range handling in bash glob patterns is mostly random, so can't be used here.

You may also want to consider what to do with non-characters, private use characters, characters that are not currently assigned in the version of Unicode your system is working with. The notion of control character is also quite fuzzy when it comes to Unicode. Are the U+202E RIGHT-TO-LEFT OVERRIDE character, the U+FEFF BOM character, the tag characters for instance control characters?

5
  • does LC_CTYPE=en_US.utf8 at the start of the script ensure that the loale is set to UTF-8? Also these not-rejected codepoints abover 0x110000 are never ccontrol characters, or are they? – matthias_buehlmann Mar 1 at 12:58
  • @matthias_buehlmann, only on systems that have such a locale and if $LC_ALL is not otherwise also set. Setting LC_CTYPE without also setting the other LC_* variables is not always well supported. Note that unless you export that variable, it will not affect commands that are executed (as opposed to those that are internal to the shell. It will not affect expr for instance above). Several systems now ship a C.UTF-8 locale which may be a better option (export LC_ALL=C.UTF-8, possibly only set locally to a function for instance). – Stéphane Chazelas Mar 1 at 13:05
  • Those codepoints above 0x110000 are not valid. How the bytes that form the would-be-UTF-8-encoding of those codepoints are interpreted by applications is unspecified and will vary between applications as you're dealing with invalid text. – Stéphane Chazelas Mar 1 at 13:08
  • @matthias_buehlmann: If you encounter them treat them as private use characters or reject the bytestream. If they become valid, having previously treated them as private use should be correct. – Joshua Mar 1 at 20:45
  • @Joshua, depends why the OP wants to detect those. For instance, a python3 script (at least with python 3.8.5 on Ubuntu) won't even start if it has a byte sequence that is the encoding of code points over 0x110000 in its arguments. – Stéphane Chazelas Mar 1 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.