77

I have several .htm files which open in Gedit without any warning/error, but when I open these same files in Jedit, it warns me of invalid UTF-8 encoding...

The HTML meta tag states "charset=ISO-8859-1". Jedit allows a List of fallback encodings and a List of encoding auto-detectors (currently "BOM XML-PI"), so my immediate problem has been resolved. But this got me thinking about: What if the meta data wasn't there?

When the encoding information is just not available, is there a CLI program which can make a "best-guess" of which encodings may apply?

And, although it is a slightly different issue; is there a CLI program which tests the validity of a known encoding?

2

4 Answers 4

106

The file command makes "best-guesses" about the encoding.

Here demonstrated on a file containing a german umlaut encoded in utf-8:

$ file umlaut-utf8.txt
umlaut-utf8.txt:     UTF-8 Unicode text

And the same umlaut in two other encodings:

$ file umlaut-iso88591.txt umlaut-utf16.txt
umlaut-iso88591.txt: ISO-8859 text
umlaut-utf16.txt:    Little-endian UTF-16 Unicode text, with no line terminators

And all three mashed together for an invalid encoding:

$ file umlaut-mixed.txt
umlaut-mixed.txt:    data

You can use the -i parameter to output in mime type:

$ file -i *
umlaut-iso88591.txt: text/plain; charset=iso-8859-1
umlaut-mixed.txt:    application/octet-stream; charset=binary
umlaut-utf16.txt:    text/plain; charset=utf-16le
umlaut-utf8.txt:     text/plain; charset=utf-8

(on mac it is -I. because apple devs think different.)

The file command is quite limited. It looks over some of the bytes and tries to guess what the encoding might be. If it recognizes a pattern it will say that it is this or that encoding. If it does not recognize a pattern, or if the recognized patterns contradict each other, it will say "data" (or binary in mime type). Which practically means no valid encoding recognized.

This is similar to how you might be able to recognize a text as being spanish or french based on the distribution of characters and umlauts. If you were given a text where the distribution of characters makes no sense then you might conclude that it is an "invalid" text. But it might be a language you just haven't seen before. Compare this to Lorem Ipsum. A text made to look like a natural text but is actually nonsense: https://en.wikipedia.org/wiki/Lorem_ipsum

Here is an example where file was not able to recognize the correct encoding: view file containing DOS text (box-drawing characters, CRLF line terminators) and escape sequences

Here is more information about the file command: http://www.linfo.org/file_command.html


How I created the files:

$ echo ä > umlaut-utf8.txt

You can copy this line and run it. It should create a file containing the umlaut in utf8.

Check the hex dump:

$ hexdump -C umlaut-utf8.txt 
00000000  c3 a4 0a                                          |...|
00000003

Convert to the other encodings:

$ iconv -f utf8 -t iso88591 umlaut-utf8.txt > umlaut-iso88591.txt 
$ iconv -f utf8 -t utf16 umlaut-utf8.txt > umlaut-utf16.txt 

The hex dumps:

$ hexdump -C umlaut-iso88591.txt 
00000000  e4 0a                                             |..|
00000002
$ hexdump -C umlaut-utf16.txt 
00000000  ff fe e4 00 0a 00                                 |......|
00000006

Compare with https://en.wikipedia.org/wiki/Ä#Computer_encoding

Create something "invalid" by mixing all three:

$ cat umlaut-iso88591.txt umlaut-utf8.txt umlaut-utf16.txt > umlaut-mixed.txt 
2
  • Thanks, that worked... I had tried 'file, but without any option :( ... I've now also tried a mixof UTF-16 and UTF-8 and ISO-8859-1. file -i` reported unknown-8bit. So, this also seems to be the answer to: "How to detect an invalid/unknown encoding"
    – Peter.O
    Apr 19, 2011 at 9:21
  • 2
    For those who get here and are on mac, it's file -I with a capital 'i' instead of lowercase. Jul 17, 2019 at 14:12
29

It isn't always possible to find out for sure what the encoding of a text file is. For example, the byte sequence \303\275 (c3 bd in hexadecimal) could be ý in UTF-8, or ý in latin1, or Ă˝ in latin2, or in BIG-5, and so on.

Some encodings have invalid byte sequences, so it's possible to rule them out for sure. This is true in particular of UTF-8; most texts in most 8-bit encodings are not valid UTF-8. You can test for valid UTF-8 with isutf8 from moreutils or with iconv -f utf-8 -t utf-8 >/dev/null, amongst others.

There are tools that try to guess the encoding of a text file. They can make mistakes, but they often work in practice as long as you don't deliberately try to fool them.

  • file
  • Perl Encode::Guess (part of the standard distribution) tries successive encodings on a byte string and returns the first encoding in which the string is valid text.
  • Enca is an encoding guesser and converter. You can give it a language name and text that you presume is in that language (the supported languages are mostly East European languages), and it tries to guess the encoding.

If there is metadata (HTML/XML charset=, TeX \inputenc, emacs -*-coding-*-, …) in the file, advanced editors like Emacs or Vim are often able to parse that metadata. That's not easy to automate from the command line though.

1
  • Thanks for the good overview... Yes, "best-guess" can be the only option when the encoding is not known... Using iconv, I just ran all 1168 encodings (including aliases) listed by iconv -l against one of my .htm files... There were 683 encodings which passed muster.. The file's actual charset=ISO-8859-1 ..made up of all bar one ASCII-range values.. The non-ASCII char was \xA9 .
    – Peter.O
    Apr 19, 2011 at 23:02
5

I think this CLI tool might do exactly what you were looking for!

Make sure you have Node.js and NPM installed:

$ sudo apt install nodejs npm

Install the CLI:

$ npm install -g detect-file-encoding-and-language

Use the CLI:

$ dfeal "/home/user name/Documents/subtitle file.srt"
# Possible result: { language: french, encoding: CP1252, confidence: 0.99 }
3
  • No offense, but this is the unix & linux stack, why would I install nodejs and npm to them install a detection program? The first answer used built in unix utils...
    – number9
    Mar 24, 2021 at 18:59
  • 4
    Well, the first answer suggests using "file -i" which unfortunately isn't always very accurate. So if someone is searching for an alternative it might be worth the extra effort of installing Node.js and NPM...
    – Falaen
    Mar 25, 2021 at 8:34
  • @number9 SE has always (AFAIK) been all about "give all valid possible answers" (their wording might differ cuz I read it years ago last, but mean the same). This is exactly that. Dec 11, 2022 at 21:46
3

Also in case you file -i gives you unknown

You can use this php command that can guess charset like below :

In php you can check like below :

Specifying encoding list explicitly :

php -r "echo 'probably : ' . mb_detect_encoding(file_get_contents('myfile.txt'), 'UTF-8, ASCII, JIS, EUC-JP, SJIS, iso-8859-1') . PHP_EOL;"

More accurate "mb_list_encodings":

php -r "echo 'probably : ' . mb_detect_encoding(file_get_contents('myfile.txt'), mb_list_encodings()) . PHP_EOL;"

Here in first example, you can see that i put a list of encodings (detect list order) that might be matching. To have more accurate result you can use all possible encodings via : mb_list_encodings()

Note mb_* functions require php-mbstring

apt-get install php-mbstring 

See answer : https://stackoverflow.com/a/57010566/3382822

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .