Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have several .htm files which open in Gedit without any warning/error, but when I open these same file in Jedit, it warns me of invalid UTF-8 encoding...

The html meta tag states "charset=ISO-8859-1"
Jedit allows a List of fallback encodings and a List of encoding auto-detectors (currently "BOM XML-PI"), so my immediate problem has been resolved. but this got me thinking about: "What if the meta data wasn't there?"

When the encoding info is just not available, is there a CLI program which can make a "best-guess" of which encodings may apply?

And, although it is a slightly differnt issue; is there a CLI program which tests the validity of a known encoding?

share|improve this question
    
Similar to "How to auto detect text file encoding?" superuser.com/questions/301552/… –  buzz3791 Jun 16 at 16:30

2 Answers 2

up vote 5 down vote accepted

the file command makes "best-guesses" about the encoding. Use the -i parameter to force file to print information about the encoding.

I created two files containing german umlauts. one encoded in utf-8 and one encoded in iso-8859-1.

$ file -i *
file1: text/plain; charset=utf-8
file2: text/plain; charset=iso-8859-1
share|improve this answer
    
Thanks, that worked... I had tried 'file`, but without any option :( ... I've now also tried a mixof UTF-16 and UTF-8 and ISO-8859-1. file -i reported unknown-8bit. So, this also seems to be the answer to: "How to detect an invalid/unknown encoding" –  Peter.O Apr 19 '11 at 9:21

It isn't always possible to find out for sure what the encoding of a text file is. For example, the byte sequence \303\275 (c3 bd in hexadecimal) could be ý in UTF-8, or ý in latin1, or Ă˝ in latin2, or in BIG-5, and so on.

Some encodings have invalid byte sequences, so it's possible to rule them out for sure. This is true in particular of UTF-8; most texts in most 8-bit encodings are not valid UTF-8. You can test for valid UTF-8 with isutf8 from moreutils or with iconv -f utf-8 -t utf-8 >/dev/null, amongst others.

There are tools that try to guess the encoding of a text file. They can make mistakes, but they often work in practice as long as you don't deliberately try to fool them.

  • file
  • Perl Encode::Guess (part of the standard distribution) tries successive encodings on a byte string and returns the first encoding in which the string is valid text.
  • Enca is an encoding guesser and converter. You can give it a language name and text that you presume is in that language (the supported languages are mostly East European languages), and it tries to guess the encoding.

If there is metadata (HTML/XML charset=, TeX \inputenc, emacs -*-coding-*-, …) in the file, advanced editors like Emacs or Vim are often able to parse that metadata. That's not easy to automate from the command line though.

share|improve this answer
    
Thanks for the good overview... Yes, "best-guess" can be the only option when the encoding is not known... Using iconv, I just ran all 1168 encodings (including aliases) listed by iconv -l against one of my .htm files... There were 683 encodings which passed muster.. The file's actual charset=ISO-8859-1 ..made up of all bar one ASCII-range values.. The non-ASCII char was \xA9 . –  Peter.O Apr 19 '11 at 23:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.