I have a very long text file in french language that I need to clean up. The non ASCII characters have been replaced by combination of odd characters. As an example, the following content :

passer de très bonnes fêtes de fin d'année.

should become : (as Unicode text)

passer de très bonnes fêtes de fin d'année.

I have tried sed, but no success. A friend recommended to try Perl. I can easily build a table with the odd sequence of characters and the correct replacing ones. Ideally I would prefer this table to be an independant file for future use. What is the recommended approach for such conversions?

  • It look like you had the text encoded in utf-8 (that is good, as it is the standard for Unix), but then something read it as ISO 8859-1 / Microsoft® Windows Latin-1 and then output it's interpretation. You need to reverse this. – ctrl-alt-delor Dec 22 '14 at 17:37
  • I am afraid that playing with standard encodings is not sufficient, as the damage is more complicated. I have tried lots of variants of iconv without luck. However, using sed was finally a way to solve the first issue: cat file.txt | sed 's/\303\203\302\250/\303\250/g' > NewFile.txt. I am trying to pipe several such commands, but so far I am getting the error: "invalid backreference number" – Yves Dec 22 '14 at 17:46
  • I think I did it with iconv (just before reading your comment). – ctrl-alt-delor Dec 22 '14 at 17:49
  • I am of course interested in your iconv command. I am far from being an expert neiter in sed, nor in iconv or other tools. – Yves Dec 22 '14 at 17:51

It look like you had the text encoded in utf-8 (that is good, as it is the standard for Unix), but then something read it as ISO 8859-1 / Microsoft®'s windows Latin-1 and then output its interpretation. You need to reverse this.


echo "passer de très bonnes fêtes de fin d'année" | iconv --to-code="ISO 8859-1"

This will take the broken encoding, and convert it to valid utf-8. If your system is configured to utf-8, then it will read correctly.

Explication: If we do echo è | od -t x1 and echo ê | od -t x1, then we see that the hex codes are c3 a8 0a and c3 aa 0a, we then look here http://www.ascii-code.com/ ( these are iso 8859-1 codes, not ascii ) we see that these codes give è and ê both followed by an invisible character. So now we know what went wrong: something read utf-8, but interpreted it as iso 8859-1. So we now need to reverse it: We read in what ever format it is that we are reading in, and convert to iso 8859-1 (the reverse of what got us here). The result is valid utf-8.

| improve this answer | |
  • Does this work for you? – ctrl-alt-delor Dec 22 '14 at 18:20
  • It did not exactly, as my version of icon (Max OSX) does not seem to handle this chnge. But your investigation led me to a solution using sed, finally. – Yves Dec 23 '14 at 8:16
  • Wrong, misleading answer. Various “©” stuff indicates that, during erroneous processing, Windows-1252 (a.k.a. Latin-1) was assumed as the input encoding, that is not exactly the same as ISO 8859-1.                  A correct solution should employ --to-code="Windows-1252", indeed. – Incnis Mrsi Sep 4 '15 at 20:16
  • In fact, Ī̲ was wrong, myself. These are ISO 8859 characters. Sorry. – Incnis Mrsi Sep 4 '15 at 22:56
  • Does this work on other encoding? If I try echo "D’²D’²" | iconv --to-code="SJIS" I get iconv: illegal input sequence at position 0 – Oni Mar 15 at 12:26

The mojibake presented by the original poster may have two causes:

  1. The file contains a French text in UTF, but the program that shows it erroneously assumes ISO 8859-1 or Windows-1252 encoding (instead of UTF-8).
  2. Double encoding: essentially the thing told by richard.

Solutions are:

  1. Read by an application with a UTF-8 locale.
  2. try iconf -f UTF-8 -t Windows-1252 <garbage_file          or iconf -f UTF-8 -t ISO-8859-1 <garbage_file      , expecting a meaningful UTF-8 text on the output.

It is possible, though, that the text was further damaged (for experts: e.g. U+00C7 Ç, UTF-encoded as \303\207, was read in ISO-8859-1 with a C1 control code, that was dropped) and automatic conversion back is no longer possible. Then yes, automatic search and replace (see comments under richard’s answer) can recover at least some original characters.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.