In Unicode, some character combinations have more than one representation.

For example, the character ä can be represented as

  • "ä", that is the codepoint U+00E4 (two bytes c3 a4 in UTF-8 encoding), or as
  • "ä", that is the two codepoints U+0061 U+0308 (three bytes 61 cc 88 in UTF-8).

According to the Unicode standard, the two representations are equivalent but in different "normalization forms", see UAX #15: Unicode Normalization Forms.

The unix toolbox has all kinds of text transformation tools, sed, tr, iconv, Perl come to mind. How can I do quick and easy NF conversion on the command-line?

up vote 18 down vote accepted

You can use the uconv utility from ICU. Normalization is achieved through transliteration (-x).

$ uconv -x any-nfd <<<ä | hd
00000000  61 cc 88 0a                                       |a...|
00000004
$ uconv -x any-nfc <<<ä | hd
00000000  c3 a4 0a                                          |...|
00000003

On Debian, Ubuntu and other derivatives, uconv is in the libicu-dev package. On Fedora, Red Hat and other derivatives, and in BSD ports, it's in the icu package.

  • This works, thanks. You have to install a 30M dev library alongside it though. What's worse, I haven't been able to find proper documentation for uconv itself: where did you find any-nfd? It looks like development of this tool has been abandoned, last update was in 2005. – glts Sep 14 '13 at 16:07
  • 1
    @glts I found any-nfd by browsing through the list displayed by uconv -L. – Gilles Sep 14 '13 at 23:38
  • On Ubuntu using sudo apt install icu-devtools to run uconv -x any-nfc, but not solve the simplest problem, e.g. a bugText.txt file with "Iglésias, Bad-á, Good-á" converted by uconv -x any-nfc bugText.txt > goodText.txt stay the same text. – Peter Krauss Nov 16 at 11:40

Python has unicodedata module in its standard library, which allow to translate Unicode representations through unicodedata.normalize() function:

import unicodedata

s1 = 'Spicy Jalape\u00f1o'
s2 = 'Spicy Jalapen\u0303o'

t1 = unicodedata.normalize('NFC', s1)
t2 = unicodedata.normalize('NFC', s2)
print(t1 == t2) 
print(ascii(t1)) 

t3 = unicodedata.normalize('NFD', s1)
t4 = unicodedata.normalize('NFD', s2)
print(t3 == t4)
print(ascii(t3))

Running with Python 3.x:

$ python3 test.py
True
'Spicy Jalape\xf1o'
True
'Spicy Jalapen\u0303o'

Python isn't well suited for shell one liners, but it can be done if you don't want to create external script:

$ python3 -c $'import unicodedata\nprint(unicodedata.normalize("NFC", "ääääää"))'
ääääää

For Python 2.x you have to add encoding line (# -*- coding: utf-8 -*-) and mark strings as Unicode with u character:

$ python -c $'# -*- coding: utf-8 -*-\nimport unicodedata\nprint(unicodedata.normalize("NFC", u"ääääää"))'
ääääää

Check it with the tool hexdump:

echo  -e "ä\c" |hexdump -C 

00000000  61 cc 88                                          |a..|
00000003  

convert with iconv and check again with hexdump:

echo -e "ä\c" | iconv -f UTF-8-MAC -t UTF-8 |hexdump -C

00000000  c3 a4                                             |..|
00000002

printf '\xc3\xa4'
ä
  • 2
    This only works on macOS. There is no 'utf-8-mac' on Linux, on FreeBSDs, etc. Also, decomposition by using this encoding does not follow the specification (it does follow the macOS filesystem normalization algorithm though). More info: search.cpan.org/~tomita/Encode-UTF8Mac-0.04/lib/Encode/… – antonone Feb 14 '17 at 11:56
  • @antonone to be fair though there was no OS specified in the question. – roaima Sep 15 '17 at 7:47
  • 1
    @roaima Yes, that's why I've assumed that the answer should work on all systems that are based on Unix/Linux. The answer above works only on macOS. If one's looking for a macOS-specific answer, then it'll work, in part. I just wanted to point that out, because the other day I've lost some time wondering why I have no utf-8-mac on Linux and if this is normal. – antonone Sep 15 '17 at 10:55

coreutils has a patch to get a proper unorm. works fine for me on 4byte wchars. follow http://crashcourse.housegordon.org/coreutils-multibyte-support.html#unorm The remaining problem there are 2-byte wchar systems (cygwin, windows, plus aix and solaris on 32bit), which need to transform codepoints from upper planes into surrogate pairs and vice versa, and the underlying libunistring/gnulib cannot handle that yet.

perl has the unichars tool, which also does the various normalization forms on the cmdline. http://search.cpan.org/dist/Unicode-Tussle/script/unichars

There's a perl utility called Charlint available from

https://www.w3.org/International/charlint/

which does what you want. You'll also have to download a file from

ftp://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt

After the first run you'll see Charlint complaining about incompatible entries in that file so you'll have to delete those lines from UnicodeData.txt.

For completeness, with perl:

$ perl -CSA -MUnicode::Normalize=NFD -e 'print NFD($_) for @ARGV' $'\ue1' | uconv -x name
\N{LATIN SMALL LETTER A}\N{COMBINING ACUTE ACCENT}
$ perl -CSA -MUnicode::Normalize=NFC -e 'print NFC($_) for @ARGV' $'a\u301' | uconv -x name
\N{LATIN SMALL LETTER A WITH ACUTE}

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.