45

I have a file in UTF-8 that contains texts in multiple languages. A lot of it are people's names. I need to convert it to ASCII and I need the result to look as decent as possible.

There are many ways how to approach converting from a wider encoding to a narrower one. The simplest transformation would be to replace all non-ASCII characters with some placeholder, like '_'. If I know the language the file is written in, there are additional possibilities, like romanization.

What Unix tool or programming language library available on Unix can give me a decent (best-effort) conversion from UTF-8 to ASCII?

Most of the text is in European, latin type based languages.

Improve this question
7
  • 1
    do you know where which language starts? There is e.g. a difference on how to handle non-availability of an umlaut (as on the ö). In German you can always write "oe", but e.g. in Dutch the unavailability of an umlaut can better be "described" by a dash followed by the umlauted character (and there the "oe" would be a completely different diphthong)
    – Anthon
    Commented Dec 6, 2014 at 17:03
  • How do you define “as decent as possible”? The real difficulty is in defining the mappings. Compared to that, the programming task is trivial. The mappings actually used vary a lot and may be language-specific in two ways: they depend on the language of the text and on the assumed language of the reader (especially as regards to romanization).
    – Jukka K. Korpela
    Commented Dec 6, 2014 at 22:54
  • 1
    @user7610 Other than iconv and tr, there is Unidecode. I am not familiar with it, but it might do what you want, if you can use Python.
    – yellowantphil
    Commented Dec 7, 2014 at 23:56
  • 1
    @yellowantphil or node-unidecode in JavaScript/node, UnidecodeSharp in C♯, or Text::Unidecode in Perl, which happens to be first of this name. I guess there are other versions.
    – user7610
    Commented Dec 8, 2014 at 11:27
  • 1
    An alternative translit command is suggested at askubuntu.com/a/1132121
    – Nemo
    Commented Apr 25, 2019 at 16:37

5 Answers 5

Reset to default
49

This will work for some things:

iconv -f utf-8 -t ascii//TRANSLIT

echo ĥéĺłœ π | iconv -f utf-8 -t ascii//TRANSLIT returns helloe ?. Any characters that iconv doesn’t know how to convert will be replaced with question marks.

iconv is POSIX, but I don’t know if all systems have the TRANSLIT option. It works for me on Linux. Also, the IGNORE option will silently discard characters that cannot be represented in the target character set (see man iconv_open).

An inferior but POSIX-compliant option is to use tr. This command replaces all non-ASCII code points with a question mark. It reads UTF-8 text one byte at a time. “É” might be replaced with E? or ?, depending on whether it was encoded using a combining accent or a precomposed character.

echo café äëïöü | tr -d '\200-\277' | tr '\300-\377' '[?*]'

That example returns caf? ?????, using precomposed characters.

Improve this answer
7
  • 1
    tr is not meant to work one byte at a time. GNU tr does, but it's a bug.
    – Stéphane Chazelas
    Commented Oct 9, 2015 at 15:46
  • 4
    iconv -f utf-8 -t ascii//TRANSLIT worked well for me. It changed curly quotes to straight quotes. Thanks.
    – Colonel Panic
    Commented Mar 30, 2016 at 15:37
  • 1
    Note that iconv will choke on heavily accented characters such as Pinyin.
    – sventechie
    Commented Dec 29, 2016 at 19:10
  • 1
    Note that //TRANSLIT also works for other sets of characters, e.g. iso-8859-1//TRANSLIT.
    – Skippy le Grand Gourou
    Commented Jun 19, 2018 at 14:33
  • 3
    -c option to iconv silentlyl discards characters that cannot be converted instead of terminating
    – mykel
    Commented May 26, 2021 at 4:16
20 
konwert utf8-ascii

It will do best-effort conversion, depending on the conversion tables. If you know approximately the input language, there are language specific filters giving better results, e.g.

konwert utf8-xmetodo

is the conversion of Esperanto into the x-metodo representation,

konwert UTF8-tex

will try to do TeX representation of diacritics, there are language specific parameters:

konwert UTF8-ascii/de

will transliterate "ä" into "ae" (customary for German) instead of plain "a"

konwert UTF8-ascii/rosyjski

will use Polish rules for transliterating Russian, instead of the "English-like" ones, etc...

Improve this answer
3
7

try uni2ascii -B input.txt >output.txt

uni2ascii

Improve this answer
3
  • Worked well for French on MacOS.
    – Andriy Makukha
    Commented Feb 15, 2020 at 17:45
  • ok, but what is the command line if someone wants to convert multiple text files from a particular folder? Can be converted from UNI to ASCII in the same files?
    – Just Me
    Commented Sep 24, 2022 at 12:42
  • @JustMe for i in `ls somedir/`; do uni2ascii -B "$i" > "$i.mod"; done
    – M Johnson
    Commented Dec 15, 2023 at 1:23
4

I ended up using Perl with Text::Unidecode for this. There are ports of the library for other languages.

Examples of a few difficult cases:

perl -e 'use utf8; use Text::Unidecode; print unidecode("عبد الله الثاني بسين")

produces bd llh lthny bn lHsyn, which is acceptable result for my purposes.

It can even do Chinese characters, to some degree:

$ perl -e 'use utf8; use Text::Unidecode; print unidecode("工廠")'
Gong Chang
$ perl -e 'use utf8; use Text::Unidecode; print unidecode("工厂")'
Gong Han
Improve this answer
1

I have a file in UTF-8 that contains [people's names] in multiple languages [that I want to convert to something meaningfull in ASCII].

You mean you want to be able to convert the following names into some ASCII string the person concerned would not object to?

  • ஸ்றீனிவாஸ ராமானுஜன் ஐயங்கார்
  • عبد الله الثاني بن الحسين

I suspect there is no automated tool that can do this. There can be either no or very many Latinizations of personal names. Software cannot choose the culturally acceptable version. At least not without the software knowing a lot about the culture of the person involved.

See also https://stackoverflow.com/a/1398403/477035

Improve this answer
2
  • 2
    perl -e 'use utf8; use Text::Unidecode; print unidecode("عبد الله الثاني بسين")' produces ``bd llh lthny bn lHsyn` which is good enough transliteration for my purposes.
    – user7610
    Commented Sep 2, 2015 at 15:53
  • 4
    @user7610: Fine but King Abdulla II of Jordan might disagree. I would prepare an explanation in case someone important complains to the CEO :-)
    – RedGrittyBrick
    Commented Sep 2, 2015 at 15:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .