1

I have a file with precomposed unicode characters. Here's wikipedia page about it. I have got a word like "Åström", but accents aren't in the alphabet of this language. I want to get rid of them, but don't know how - I've tried copying and pasting the characters(worked for some before - probably they weren't composed), but this doesn't give the expected result.

2
  • What is your desired output? Decomposed characters? Just the base characters with accents stripped? Feb 11, 2016 at 10:56
  • My goal is to find base characters without accents, yes. I want to get rid of accents. Feb 11, 2016 at 10:57

3 Answers 3

2

Your Åström has decomposed unicode characters, not precomposed ones:

(I'm assuming the current locale uses UTF-8 and those characters entered on the command line use UTF-8 encoding here (I've copy-pasted them from a browser using UTF-8)).

$ printf %s Åström | uconv -x any-name
\N{LATIN CAPITAL LETTER A}\N{COMBINING RING ABOVE}\N{LATIN SMALL LETTER S}\N{LATIN SMALL LETTER T}\N{LATIN SMALL LETTER R}\N{LATIN SMALL LETTER O}\N{COMBINING DIAERESIS}\N{LATIN SMALL LETTER M}

Note the Combining diacritics above. Pre-composed ones would be:

$ printf %s Åström | uconv -x nfkc | uconv -x any-name
\N{LATIN CAPITAL LETTER A WITH RING ABOVE}\N{LATIN SMALL LETTER S}\N{LATIN SMALL LETTER T}\N{LATIN SMALL LETTER R}\N{LATIN SMALL LETTER O WITH DIAERESIS}\N{LATIN SMALL LETTER M}

Those pre-composed characters, contrary to the combining accents are present in iso8859-1 or iso8859-15. So if that's the intended destination character set, you could do:

$ printf %s Åström | uconv -x nfkc -t iso-8859-1

To convert the characters that can be converted, approximate the other ones:

$ printf %s 'Åström й' | uconv -x nfkc | iconv -t iso-8859-5//TRANSLIT
Astrom <0xd9>

(й is in the iso-8859-5 charset, so is converted to its representation there (0xd9 byte), but not Å, or ö which are then converted to a A and o approximation).

If you mean that your input has a combination of decomposed characters (like those and ) and pre-composed ones (like й) and you want to keep the precomposed ones, but discard the combining characters in the decomposed ones, then you can do:

$ printf %s 'Åström й' | uconv -x '[:Nonspacing Mark:]>'
Astrom й

Note that there are two characters in Unicode with a Å visual representation: U+212B (Ångström) and U+00C5 (A with ring above). nfkc will convert U+212B to U+00C5.

If you want to remove all the diacritics, you can try:

$ printf %s Åström | iconv -t us//TRANSLIT
Astrom

(beware that some implementations of iconv may approximate the accents with adjacent " or '... characters)

Or:

$ printf %s Åström | uconv -x nfd -c -t us
Astrom

(decompose, convert to ASCII, dropping characters that can't be converted like combining accents).

Or:

$ printf %s Åström | uconv -x "::nfd;[:Nonspacing Mark:]>;"
Astrom

(decompose, drop all non-spacing marks)

Or:

$ printf %s Åström | uconv -x Latin-ASCII
Astrom

(ASCII transliteration for characters in the Latin script).

uconv is an utility from the ICU Project. On Debian and derivatives, you find it in the icu-devtools package.

9
  • What happened here? I see the ring over the s and the umlaut above the m. Is my browser wrong, or do others see it as well? (Using Firefox 44.0 on Ubuntu 14.04 LTS) Feb 11, 2016 at 11:15
  • Thanks, but I don't want to remove diacritics; actually I don't know what diacritics would be, since I'm dealing with Cyrillic - so и and й are fine and different letters, but и with ` above it would be wrong. Feb 11, 2016 at 11:17
  • @jknappen, that's a bug in your browser or rendering library (my browser (iceweasel on Debian) has the same bug, chromium doesn't). Feb 11, 2016 at 11:37
  • @MatthewRock, you need to clarify what is fine and what isn't. й is in iso-8859-5, is that the charset your characters need to be present in? Would iconv -t nfc -t iso-8859-5 work for you? Feb 11, 2016 at 11:41
  • Yeah, I might have been unclear. I am using utf-8, and I want Cyrillic alphabet. Apart from that, I don't have much more requirements. Feb 11, 2016 at 12:32
1

There is a perl module Text::Unaccent available from CPAN for your purpose:

http://search.cpan.org/~ldachary/Text-Unaccent-1.08/Unaccent.pm

A sample perl script (working from STDIN to STDOUT) may look like:

#!/usr/bin/perl
use Text::Unaccent;
while ($zeile = <STDIN>)
{
   print STDOUT unac_string("UTF-8", $zeile);
}
4
  • How can I get it and use it? Feb 11, 2016 at 11:37
  • There is a download link in the upper right part of the page I quoted. Feb 11, 2016 at 13:00
  • @MatthewRock: To use it, write a small perl script reading your input file line by line, converting the line to unaccented and write them out again. Feb 11, 2016 at 13:01
  • I might have been unclear - I am completely unfluent in Perl. Now since I am a programmer it's not an issue for me to do 10-minute research to write this script, but right now your answer is link-only - providing code snippet would both make it full answer, and help others who are not programmers, but stumble upon similar problem. Feb 11, 2016 at 13:12
-1

The solution I used is to use Emacs. I open the file, find the part with accents, copy a character with accent and one character before the accent. I go to the beginning of file, run M-x replace-string, paste what I copied, go to the beginning of the minibuffer, delete the character that was before, and run the command.

I figured that the accent was more like a pre-character, so by copying both the character I wanted and character before, I ensured to copy everything I wanted.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .