13

While trying to convert a text file into its ASCII equivalent, I get error message that iconv: illegal input sequence at position.

Command I use is iconv -f UTF-8 -t ascii//TRANSLIT file

The offending character is æ.

Text file itself is present here.

Why does it say illegal sequence? The input character is proper UTF-8 character (U+00E6).

16

The file is encoded in ISO-8859-1, not in UTF-8:

$ hd 0606461.txt | grep -B1 '^0002c520'
0002c510  64 75 6d 20 66 65 72 69  65 6e 74 20 72 75 69 6e  |dum ferient ruin|
0002c520  e6 0d 0a 2d 2d 48 6f 72  61 63 65 2e 0d 0a 0d 0a  |...--Horace.....|

And the byte "e6" alone is not a valid UTF-8 sequence.

So, use iconv -f latin1 -t ascii//TRANSLIT file.

  • How do find out which is the byte corresponding to offending character? I tried hexdump -C file command and got 0002b220 72 75 69 6e e6 0a 20 2d 2d 20 48 6f 72 61 63 65 |ruin.. -- Horace| as output. – user13107 Jul 9 '14 at 23:12
  • 1
    In what you got, you can see that the only top-bit-set byte (a byte whose value is ≥ 80 in hexadecimal) is e6. This doesn't correspond to a valid UTF-8 sequence (in UTF-8, non-ASCII characters need at least 2 top-bit-set bytes). In ISO-8859-1, e6 is the encoding of the character "æ", which corresponds to the expected text; so, this confirms that the ISO-8859-1 encoding (or similar) is used for this file. – vinc17 Jul 9 '14 at 23:41
5

The file you linked appears to be UTF-8 inside an HTML document

$ file 0606461.txt 
0606461.txt: HTML document, ASCII text, with CRLF line terminators

If you run it through an HTML-to-text converter first, e.g.

iconv -f UTF-8 -t ascii//TRANSLIT < <(html2text 0606461.txt)

then the UTF-8 fragment you appear to be having trouble with appears to transliterate without error i.e

Si fractus illabatur orbis.
Impavidum ferient ruinæ
--Horace.

becomes

Si fractus illabatur orbis.
Impavidum ferient ruinae
--Horace.

The html2text utility may not be installed on your system - if you can't locate/install it there are other converters including a python module.

  • No, the file is not encoded in UTF-8, but in ISO-8859-1. BTW, the file command says ASCII, but the reason is that it just looks at the beginning of the file, and the ISO-8859-1 character appears far away, at position 181536. – vinc17 Jul 9 '14 at 15:19
  • @vinc17 how did you find out the file was in ISO-8859? – user13107 Jul 9 '14 at 22:18
  • 1
    @user13107 by looking at the encoding of the offending character: it is the byte "e6", not the UTF-8 sequence "c3 a6". Emacs also detected the file as being in ISO-8859-1. – vinc17 Jul 9 '14 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.