5

I have a very long text file in french language that I need to clean up. The non ASCII characters have been replaced by combination of odd characters. As an example, the following content :

passer de très bonnes fêtes de fin d'année.

should become : (as Unicode text)

passer de très bonnes fêtes de fin d'année.

I have tried sed, but no success. A friend recommended to try Perl. I can easily build a table with the odd sequence of characters and the correct replacing ones. Ideally I would prefer this table to be an independant file for future use. What is the recommended approach for such conversions?

4
  • 1
    It look like you had the text encoded in utf-8 (that is good, as it is the standard for Unix), but then something read it as ISO 8859-1 / Microsoft® Windows Latin-1 and then output it's interpretation. You need to reverse this. Dec 22, 2014 at 17:37
  • I am afraid that playing with standard encodings is not sufficient, as the damage is more complicated. I have tried lots of variants of iconv without luck. However, using sed was finally a way to solve the first issue: cat file.txt | sed 's/\303\203\302\250/\303\250/g' > NewFile.txt. I am trying to pipe several such commands, but so far I am getting the error: "invalid backreference number"
    – Yves
    Dec 22, 2014 at 17:46
  • I think I did it with iconv (just before reading your comment). Dec 22, 2014 at 17:49
  • I am of course interested in your iconv command. I am far from being an expert neiter in sed, nor in iconv or other tools.
    – Yves
    Dec 22, 2014 at 17:51

2 Answers 2

6

It looks like you had the text encoded in utf-8 (that is good, as it is the standard for Unix), but then something read it as ISO 8859-1 / Microsoft's windows Latin-1 and then output its interpretation. You need to reverse this.

e.g.

echo "passer de très bonnes fêtes de fin d'année" | iconv --to-code="ISO 8859-1"

This will take the broken encoding, and convert it to valid utf-8. If your system is configured to utf-8, then it will read correctly.

Explication: If we do echo è | od -t x1 and echo ê | od -t x1, then we see that the hex codes are c3 a8 0a and c3 aa 0a, we then look here http://www.ascii-code.com/ ( these are iso 8859-1 codes, not ascii ) we see that these codes give è and ê both followed by an invisible character. So now we know what went wrong: something read utf-8, but interpreted it as iso 8859-1. So we now need to reverse it: We read in what ever format it is that we are reading in, and convert to iso 8859-1 (the reverse of what got us here). The result is valid utf-8.

8
  • Does this work for you? Dec 22, 2014 at 18:20
  • It did not exactly, as my version of icon (Max OSX) does not seem to handle this chnge. But your investigation led me to a solution using sed, finally.
    – Yves
    Dec 23, 2014 at 8:16
  • Wrong, misleading answer. Various “©” stuff indicates that, during erroneous processing, Windows-1252 (a.k.a. Latin-1) was assumed as the input encoding, that is not exactly the same as ISO 8859-1.                  A correct solution should employ --to-code="Windows-1252", indeed. Sep 4, 2015 at 20:16
  • In fact, Ī̲ was wrong, myself. These are ISO 8859 characters. Sorry. Sep 4, 2015 at 22:56
  • Does this work on other encoding? If I try echo "D’²D’²" | iconv --to-code="SJIS" I get iconv: illegal input sequence at position 0
    – Oni
    Mar 15, 2020 at 12:26
1

The mojibake presented by the original poster may have two causes:

  1. The file contains a French text in UTF, but the program that shows it erroneously assumes ISO 8859-1 or Windows-1252 encoding (instead of UTF-8).
  2. Double encoding: essentially the thing told by richard.

Solutions are:

  1. Read by an application with a UTF-8 locale.
  2. try iconf -f UTF-8 -t Windows-1252 <garbage_file          or iconf -f UTF-8 -t ISO-8859-1 <garbage_file      , expecting a meaningful UTF-8 text on the output.

It is possible, though, that the text was further damaged (for experts: e.g. U+00C7 Ç, UTF-encoded as \303\207, was read in ISO-8859-1 with a C1 control code, that was dropped) and automatic conversion back is no longer possible. Then yes, automatic search and replace (see comments under richard’s answer) can recover at least some original characters.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .