How can I check in bash whether a variable contains a valid UTF-8 string without any special control characters (such as newline or backspace or carriage return etc.)?
Add a comment
|
Provided the locale uses UTF-8 as its character encoding (check with locale charmap):
[[ $string =~ ^[^[:cntrl:]]*$ ]]
should work except that at least on GNU systems, it fails to reject strings that contain the UTF-8 encoding of characters with code points above 0x110000 (which are no longer valid in the current definition of UTF-8 encoding). That's down to the system's regexp library bash uses to do pattern matching.
That's the same for GNU expr and:
expr " $string" : ' [^[:cntrl:]]*$' > /dev/null
Here, you may want to switch to zsh where, in UTF-8 locales,
set -o extendedglob
[[ $string = [' '-$'\ud7fff'$'\ue000'-$'\U10FFFF']# && $string != *[[:cntrl:]]* ]]
should work consistently regardless of the system (at least for the matching of valid characters; you might find variations between systems for which character is classified as [:cntrl:]).
In zsh glob patterns, character ranges in multibyte character locales are based on the the wide character value of the character, which in UTF-8 is invariably the Unicode code point. Bytes that don't form part of valid characters are assigned wchar_t values in the 0xDC80..0xDCFF range in the second part of the 0xD800..0xDFFF UTF16 surrogate pairs which are not characters (a common approach sanctioned by Unicode to be able to handle arbitrary sequences of bytes).
Character range handling in bash glob patterns is mostly random, so can't be used here.
You may also want to consider what to do with non-characters, private use characters, characters that are not currently assigned in the version of Unicode your system is working with. The notion of control character is also quite fuzzy when it comes to Unicode. Are the U+202E RIGHT-TO-LEFT OVERRIDE character, the U+FEFF BOM character, the tag characters for instance control characters?
answered Mar 1 at 7:02
-
does
LC_CTYPE=en_US.utf8at the start of the script ensure that the loale is set to UTF-8? Also these not-rejected codepoints abover 0x110000 are never ccontrol characters, or are they? – matthias_buehlmann Mar 1 at 12:58 -
@matthias_buehlmann, only on systems that have such a locale and if
$LC_ALLis not otherwise also set. SettingLC_CTYPEwithout also setting the otherLC_*variables is not always well supported. Note that unless you export that variable, it will not affect commands that are executed (as opposed to those that are internal to the shell. It will not affectexprfor instance above). Several systems now ship aC.UTF-8locale which may be a better option (export LC_ALL=C.UTF-8, possibly only set locally to a function for instance). – Stéphane Chazelas Mar 1 at 13:05 -
Those codepoints above 0x110000 are not valid. How the bytes that form the would-be-UTF-8-encoding of those codepoints are interpreted by applications is unspecified and will vary between applications as you're dealing with invalid text. – Stéphane Chazelas Mar 1 at 13:08
-
@matthias_buehlmann: If you encounter them treat them as private use characters or reject the bytestream. If they become valid, having previously treated them as private use should be correct. – Joshua Mar 1 at 20:45
-
@Joshua, depends why the OP wants to detect those. For instance, a
python3script (at least with python 3.8.5 on Ubuntu) won't even start if it has a byte sequence that is the encoding of code points over 0x110000 in its arguments. – Stéphane Chazelas Mar 1 at 20:50
