18

I have a file in UTF-8 that contains texts in multiple languages. A lot of it are people's names. I need to convert it to ASCII and I need the result to look as decent as possible.

There are many ways how to approach converting from a wider encoding to a narrower one. The simplest transformation would be to replace all non-ASCII characters with some placeholder, like '_'. If I know the language the file is written in, there are additional possibilities, like romanization.

What Unix tool or programming language library available on Unix can give me a decent (best-effort) conversion from UTF-8 to ASCII?

Most of the text is in European, latin type based languages.

  • 1
    do you know where which language starts? There is e.g. a difference on how to handle non-availability of an umlaut (as on the ö). In German you can always write "oe", but e.g. in Dutch the unavailability of an umlaut can better be "described" by a dash followed by the umlauted character (and there the "oe" would be a completely different diphthong) – Anthon Dec 6 '14 at 17:03
  • How do you define “as decent as possible”? The real difficulty is in defining the mappings. Compared to that, the programming task is trivial. The mappings actually used vary a lot and may be language-specific in two ways: they depend on the language of the text and on the assumed language of the reader (especially as regards to romanization). – Jukka K. Korpela Dec 6 '14 at 22:54
  • @JukkaK.Korpela "as decent as possible" is of course defined by those who created the "Unix tool or programming language library available on Unix" that I am asking for. If the best I am gonna get is replacing everything non-ASCII with an underscore, then there is not much else I can do. Except writing my own tool, which I won't. I guess Unix@SO might not be the best place for this question… – user7610 Dec 6 '14 at 23:23
  • 1
    @user7610 Other than iconv and tr, there is Unidecode. I am not familiar with it, but it might do what you want, if you can use Python. – yellowantphil Dec 7 '14 at 23:56
  • 1
    @yellowantphil or node-unidecode in JavaScript/node, UnidecodeSharp in C♯, or Text::Unidecode in Perl, which happens to be first of this name. I guess there are other versions. – user7610 Dec 8 '14 at 11:27
9
konwert utf8-ascii

It will do best-effort conversion, depending on the conversion tables. If you know approximately the input language, there are language specific filters giving better results, e.g.

konwert utf8-xmetodo

is the conversion of Esperanto into the x-metodo representation,

konwert UTF8-tex

will try to do TeX representation of diacritics, there are language specific parameters:

konwert UTF8-ascii/de

will transliterate "ä" into "ae" (customary for German) instead of plain "a"

konwert UTF8-ascii/rosyjski

will use Polish rules for transliterating Russian, instead of the "English-like" ones, etc...

20

This will work for some things:

iconv -f utf-8 -t ascii//TRANSLIT

echo ĥéĺłœ π | iconv -f utf-8 -t ascii//TRANSLIT returns helloe ?. Any characters that iconv doesn’t know how to convert will be replaced with question marks.

iconv is POSIX, but I don’t know if all systems have the TRANSLIT option. It works for me on Linux.

An inferior but POSIX-compliant option is to use tr. This command replaces all non-ASCII code points with a question mark. It reads UTF-8 text one byte at a time. “É” might be replaced with E? or ?, depending on whether it was encoded using a combining accent or a precomposed character.

echo café äëïöü | tr -d '\200-\277' | tr '\300-\377' '[?*]'

That example returns caf? ?????, using precomposed characters.

  • tr is not meant to work one byte at a time. GNU tr does, but it's a bug. – Stéphane Chazelas Oct 9 '15 at 15:46
  • 2
    iconv -f utf-8 -t ascii//TRANSLIT worked well for me. It changed curly quotes to straight quotes. Thanks. – Colonel Panic Mar 30 '16 at 15:37
  • Note that iconv will choke on heavily accented characters such as Pinyin. – sventechie Dec 29 '16 at 19:10
  • Note that //TRANSLIT also works for other sets of characters, e.g. iso-8859-1//TRANSLIT. – Skippy le Grand Gourou Jun 19 '18 at 14:33
  • iconv gives iconv: illegal input sequence at position 1234 and truncates the file for me. Would be nice if it just deleted the character and tried to pick up the sequence again. – jozxyqk Jul 18 '18 at 23:21
1

I have a file in UTF-8 that contains [people's names] in multiple languages [that I want to convert to something meaningfull in ASCII].

You mean you want to be able to convert the following names into some ASCII string the person concerned would not object to?

  • ஸ்றீனிவாஸ ராமானுஜன் ஐயங்கார்
  • عبد الله الثاني بن الحسين

I suspect there is no automated tool that can do this. There can be either no or very many Latinizations of personal names. Software cannot choose the culturally acceptable version. At least not without the software knowing a lot about the culture of the person involved.

See also https://stackoverflow.com/a/1398403/477035

  • 2
    perl -e 'use utf8; use Text::Unidecode; print unidecode("عبد الله الثاني بسين")' produces ``bd llh lthny bn lHsyn` which is good enough transliteration for my purposes. – user7610 Sep 2 '15 at 15:53
  • 4
    @user7610: Fine but King Abdulla II of Jordan might disagree. I would prepare an explanation in case someone important complains to the CEO :-) – RedGrittyBrick Sep 2 '15 at 15:56
1

I ended up using Perl with Text::Unidecode for this. Example:

perl -e 'use utf8; use Text::Unidecode; print unidecode("عبد الله الثاني بسين")

produces bd llh lthny bn lHsyn, which is acceptable result for my purposes.

0

try uni2ascii -B input.txt >output.txt

uni2ascii

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.