35

I have a file in UTF-8 that contains texts in multiple languages. A lot of it are people's names. I need to convert it to ASCII and I need the result to look as decent as possible.

There are many ways how to approach converting from a wider encoding to a narrower one. The simplest transformation would be to replace all non-ASCII characters with some placeholder, like '_'. If I know the language the file is written in, there are additional possibilities, like romanization.

What Unix tool or programming language library available on Unix can give me a decent (best-effort) conversion from UTF-8 to ASCII?

Most of the text is in European, latin type based languages.

7
  • 1
    do you know where which language starts? There is e.g. a difference on how to handle non-availability of an umlaut (as on the ö). In German you can always write "oe", but e.g. in Dutch the unavailability of an umlaut can better be "described" by a dash followed by the umlauted character (and there the "oe" would be a completely different diphthong)
    – Anthon
    Dec 6, 2014 at 17:03
  • How do you define “as decent as possible”? The real difficulty is in defining the mappings. Compared to that, the programming task is trivial. The mappings actually used vary a lot and may be language-specific in two ways: they depend on the language of the text and on the assumed language of the reader (especially as regards to romanization). Dec 6, 2014 at 22:54
  • 1
    @user7610 Other than iconv and tr, there is Unidecode. I am not familiar with it, but it might do what you want, if you can use Python. Dec 7, 2014 at 23:56
  • 1
    @yellowantphil or node-unidecode in JavaScript/node, UnidecodeSharp in C♯, or Text::Unidecode in Perl, which happens to be first of this name. I guess there are other versions.
    – user7610
    Dec 8, 2014 at 11:27
  • 1
    An alternative translit command is suggested at askubuntu.com/a/1132121
    – Nemo
    Apr 25, 2019 at 16:37

5 Answers 5

42

This will work for some things:

iconv -f utf-8 -t ascii//TRANSLIT

echo ĥéĺłœ π | iconv -f utf-8 -t ascii//TRANSLIT returns helloe ?. Any characters that iconv doesn’t know how to convert will be replaced with question marks.

iconv is POSIX, but I don’t know if all systems have the TRANSLIT option. It works for me on Linux. Also, the IGNORE option will silently discard characters that cannot be represented in the target character set (see man iconv_open).

An inferior but POSIX-compliant option is to use tr. This command replaces all non-ASCII code points with a question mark. It reads UTF-8 text one byte at a time. “É” might be replaced with E? or ?, depending on whether it was encoded using a combining accent or a precomposed character.

echo café äëïöü | tr -d '\200-\277' | tr '\300-\377' '[?*]'

That example returns caf? ?????, using precomposed characters.

7
  • 1
    tr is not meant to work one byte at a time. GNU tr does, but it's a bug. Oct 9, 2015 at 15:46
  • 4
    iconv -f utf-8 -t ascii//TRANSLIT worked well for me. It changed curly quotes to straight quotes. Thanks. Mar 30, 2016 at 15:37
  • 1
    Note that iconv will choke on heavily accented characters such as Pinyin.
    – sventechie
    Dec 29, 2016 at 19:10
  • 1
    Note that //TRANSLIT also works for other sets of characters, e.g. iso-8859-1//TRANSLIT. Jun 19, 2018 at 14:33
  • 3
    -c option to iconv silentlyl discards characters that cannot be converted instead of terminating
    – mykel
    May 26, 2021 at 4:16
18
konwert utf8-ascii

It will do best-effort conversion, depending on the conversion tables. If you know approximately the input language, there are language specific filters giving better results, e.g.

konwert utf8-xmetodo

is the conversion of Esperanto into the x-metodo representation,

konwert UTF8-tex

will try to do TeX representation of diacritics, there are language specific parameters:

konwert UTF8-ascii/de

will transliterate "ä" into "ae" (customary for German) instead of plain "a"

konwert UTF8-ascii/rosyjski

will use Polish rules for transliterating Russian, instead of the "English-like" ones, etc...

3
6

try uni2ascii -B input.txt >output.txt

uni2ascii

1
  • Worked well for French on MacOS. Feb 15, 2020 at 17:45
3

I ended up using Perl with Text::Unidecode for this. Example:

perl -e 'use utf8; use Text::Unidecode; print unidecode("عبد الله الثاني بسين")

produces bd llh lthny bn lHsyn, which is acceptable result for my purposes.

1

I have a file in UTF-8 that contains [people's names] in multiple languages [that I want to convert to something meaningfull in ASCII].

You mean you want to be able to convert the following names into some ASCII string the person concerned would not object to?

  • ஸ்றீனிவாஸ ராமானுஜன் ஐயங்கார்
  • عبد الله الثاني بن الحسين

I suspect there is no automated tool that can do this. There can be either no or very many Latinizations of personal names. Software cannot choose the culturally acceptable version. At least not without the software knowing a lot about the culture of the person involved.

See also https://stackoverflow.com/a/1398403/477035

2
  • 2
    perl -e 'use utf8; use Text::Unidecode; print unidecode("عبد الله الثاني بسين")' produces ``bd llh lthny bn lHsyn` which is good enough transliteration for my purposes.
    – user7610
    Sep 2, 2015 at 15:53
  • 4
    @user7610: Fine but King Abdulla II of Jordan might disagree. I would prepare an explanation in case someone important complains to the CEO :-) Sep 2, 2015 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.