1

I have a file with precomposed unicode characters. Here's wikipedia page about it. I have got a word like "Åström", but accents aren't in the alphabet of this language. I want to get rid of them, but don't know how - I've tried copying and pasting the characters(worked for some before - probably they weren't composed), but this doesn't give the expected result.

  • What is your desired output? Decomposed characters? Just the base characters with accents stripped? – jk - Reinstate Monica Feb 11 '16 at 10:56
  • My goal is to find base characters without accents, yes. I want to get rid of accents. – MatthewRock Feb 11 '16 at 10:57
2

Your Åström has decomposed unicode characters, not precomposed ones:

(I'm assuming the current locale uses UTF-8 and those characters entered on the command line use UTF-8 encoding here (I've copy-pasted them from a browser using UTF-8)).

$ printf %s Åström | uconv -x any-name
\N{LATIN CAPITAL LETTER A}\N{COMBINING RING ABOVE}\N{LATIN SMALL LETTER S}\N{LATIN SMALL LETTER T}\N{LATIN SMALL LETTER R}\N{LATIN SMALL LETTER O}\N{COMBINING DIAERESIS}\N{LATIN SMALL LETTER M}

Note the Combining diacritics above. Pre-composed ones would be:

$ printf %s Åström | uconv -x nfkc | uconv -x any-name
\N{LATIN CAPITAL LETTER A WITH RING ABOVE}\N{LATIN SMALL LETTER S}\N{LATIN SMALL LETTER T}\N{LATIN SMALL LETTER R}\N{LATIN SMALL LETTER O WITH DIAERESIS}\N{LATIN SMALL LETTER M}

Those pre-composed characters, contrary to the combining accents are present in iso8859-1 or iso8859-15. So if that's the intended destination character set, you could do:

$ printf %s Åström | uconv -x nfkc -t iso-8859-1

To convert the characters that can be converted, approximate the other ones:

$ printf %s 'Åström й' | uconv -x nfkc | iconv -t iso-8859-5//TRANSLIT
Astrom <0xd9>

(й is in the iso-8859-5 charset, so is converted to its representation there (0xd9 byte), but not Å, or ö which are then converted to a A and o approximation).

If you mean that your input has a combination of decomposed characters (like those and ) and pre-composed ones (like й) and you want to keep the precomposed ones, but discard the combining characters in the decomposed ones, then you can do:

$ printf %s 'Åström й' | uconv -x '[:Nonspacing Mark:]>'
Astrom й

Note that there are two characters in Unicode with a Å visual representation: U+212B (Ångström) and U+00C5 (A with ring above). nfkc will convert U+212B to U+00C5.

If you want to remove all the diacritics, you can try:

$ printf %s Åström | iconv -t us//TRANSLIT
Astrom

(beware that some implementations of iconv may approximate the accents with adjacent " or '... characters)

Or:

$ printf %s Åström | uconv -x nfd -c -t us
Astrom

(decompose, convert to ASCII, dropping characters that can't be converted like combining accents).

Or:

$ printf %s Åström | uconv -x "::nfd;[:Nonspacing Mark:]>;"
Astrom

(decompose, drop all non-spacing marks)

Or:

$ printf %s Åström | uconv -x Latin-ASCII
Astrom

(ASCII transliteration for characters in the Latin script).

uconv is an utility from the ICU Project. On Debian and derivatives, you find it in the icu-devtools package.

| improve this answer | |
  • What happened here? I see the ring over the s and the umlaut above the m. Is my browser wrong, or do others see it as well? (Using Firefox 44.0 on Ubuntu 14.04 LTS) – jk - Reinstate Monica Feb 11 '16 at 11:15
  • Thanks, but I don't want to remove diacritics; actually I don't know what diacritics would be, since I'm dealing with Cyrillic - so и and й are fine and different letters, but и with ` above it would be wrong. – MatthewRock Feb 11 '16 at 11:17
  • @jknappen, that's a bug in your browser or rendering library (my browser (iceweasel on Debian) has the same bug, chromium doesn't). – Stéphane Chazelas Feb 11 '16 at 11:37
  • @MatthewRock, you need to clarify what is fine and what isn't. й is in iso-8859-5, is that the charset your characters need to be present in? Would iconv -t nfc -t iso-8859-5 work for you? – Stéphane Chazelas Feb 11 '16 at 11:41
  • Yeah, I might have been unclear. I am using utf-8, and I want Cyrillic alphabet. Apart from that, I don't have much more requirements. – MatthewRock Feb 11 '16 at 12:32
1

There is a perl module Text::Unaccent available from CPAN for your purpose:

http://search.cpan.org/~ldachary/Text-Unaccent-1.08/Unaccent.pm

A sample perl script (working from STDIN to STDOUT) may look like:

#!/usr/bin/perl
use Text::Unaccent;
while ($zeile = <STDIN>)
{
   print STDOUT unac_string("UTF-8", $zeile);
}
| improve this answer | |
  • How can I get it and use it? – MatthewRock Feb 11 '16 at 11:37
  • There is a download link in the upper right part of the page I quoted. – jk - Reinstate Monica Feb 11 '16 at 13:00
  • @MatthewRock: To use it, write a small perl script reading your input file line by line, converting the line to unaccented and write them out again. – jk - Reinstate Monica Feb 11 '16 at 13:01
  • I might have been unclear - I am completely unfluent in Perl. Now since I am a programmer it's not an issue for me to do 10-minute research to write this script, but right now your answer is link-only - providing code snippet would both make it full answer, and help others who are not programmers, but stumble upon similar problem. – MatthewRock Feb 11 '16 at 13:12
-1

The solution I used is to use Emacs. I open the file, find the part with accents, copy a character with accent and one character before the accent. I go to the beginning of file, run M-x replace-string, paste what I copied, go to the beginning of the minibuffer, delete the character that was before, and run the command.

I figured that the accent was more like a pre-character, so by copying both the character I wanted and character before, I ensured to copy everything I wanted.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.