Replace composed unicode characters

Question

I have a file with precomposed unicode characters. Here's wikipedia page about it. I have got a word like "Åström", but accents aren't in the alphabet of this language. I want to get rid of them, but don't know how - I've tried copying and pasting the characters(worked for some before - probably they weren't composed), but this doesn't give the expected result.

Answer 1

Your Åström has decomposed unicode characters, not precomposed ones:

(I'm assuming the current locale uses UTF-8 and those characters entered on the command line use UTF-8 encoding here (I've copy-pasted them from a browser using UTF-8)).

$ printf %s Åström | uconv -x any-name
\N{LATIN CAPITAL LETTER A}\N{COMBINING RING ABOVE}\N{LATIN SMALL LETTER S}\N{LATIN SMALL LETTER T}\N{LATIN SMALL LETTER R}\N{LATIN SMALL LETTER O}\N{COMBINING DIAERESIS}\N{LATIN SMALL LETTER M}

Note the Combining diacritics above. Pre-composed ones would be:

$ printf %s Åström | uconv -x nfkc | uconv -x any-name
\N{LATIN CAPITAL LETTER A WITH RING ABOVE}\N{LATIN SMALL LETTER S}\N{LATIN SMALL LETTER T}\N{LATIN SMALL LETTER R}\N{LATIN SMALL LETTER O WITH DIAERESIS}\N{LATIN SMALL LETTER M}

Those pre-composed characters, contrary to the combining accents are present in iso8859-1 or iso8859-15. So if that's the intended destination character set, you could do:

$ printf %s Åström | uconv -x nfkc -t iso-8859-1

To convert the characters that can be converted, approximate the other ones:

$ printf %s 'Åström й' | uconv -x nfkc | iconv -t iso-8859-5//TRANSLIT
Astrom <0xd9>

(й is in the iso-8859-5 charset, so is converted to its representation there (0xd9 byte), but not Å, or ö which are then converted to a A and o approximation).

If you mean that your input has a combination of decomposed characters (like those Å and ö) and pre-composed ones (like й) and you want to keep the precomposed ones, but discard the combining characters in the decomposed ones, then you can do:

$ printf %s 'Åström й' | uconv -x '[:Nonspacing Mark:]>'
Astrom й

Note that there are two characters in Unicode with a Å visual representation: U+212B (Ångström) and U+00C5 (A with ring above). nfkc will convert U+212B to U+00C5.

If you want to remove all the diacritics, you can try:

$ printf %s Åström | iconv -t us//TRANSLIT
Astrom

(beware that some implementations of iconv may approximate the accents with adjacent " or '... characters)

Or:

$ printf %s Åström | uconv -x nfd -c -t us
Astrom

(decompose, convert to ASCII, dropping characters that can't be converted like combining accents).

Or:

$ printf %s Åström | uconv -x "::nfd;[:Nonspacing Mark:]>;"
Astrom

(decompose, drop all non-spacing marks)

Or:

$ printf %s Åström | uconv -x Latin-ASCII
Astrom

(ASCII transliteration for characters in the Latin script).

uconv is an utility from the ICU Project. On Debian and derivatives, you find it in the icu-devtools package.

Answer 2

There is a perl module Text::Unaccent available from CPAN for your purpose:

http://search.cpan.org/~ldachary/Text-Unaccent-1.08/Unaccent.pm

A sample perl script (working from STDIN to STDOUT) may look like:

#!/usr/bin/perl
use Text::Unaccent;
while ($zeile = <STDIN>)
{
   print STDOUT unac_string("UTF-8", $zeile);
}

Answer 3

The solution I used is to use Emacs. I open the file, find the part with accents, copy a character with accent and one character before the accent. I go to the beginning of file, run M-x replace-string, paste what I copied, go to the beginning of the minibuffer, delete the character that was before, and run the command.

I figured that the accent was more like a pre-character, so by copying both the character I wanted and character before, I ensured to copy everything I wanted.