1

The iconv command may change file encodings. But is there a command to find the file encoding of a certain text file. Also, I am trying to find what file encoding standards each country use. So that I may change the encoding to the correct ISO standard.

Which command may find a file encoding? .txt extension. Or others such as .py, or .c, source code files?

Then change it to the correct countries encoding. Based on a countries standard. I am trying to find references for the correct endian formats, and other relative documentation.

Making a text file created UTF-8 in America, formatted for Russian. As if it was made in Russian formatted compatibility.

7
  • 3
    file can recognise many different kinds of files, including different kinds of text files (plain ASCII, UTF-8, UTF-16, and others). It's not infallible, but it does a decent job.
    – cas
    Apr 24, 2021 at 6:00
  • 1
    I believe you need heuristics. If a certain encoding generates gibberish, it might be an incorrect encoding, or perhaps the file is supposed to contain gibberish. A very similar question, although based on Windows: softwareengineering.stackexchange.com/questions/187169/… Apr 24, 2021 at 6:03
  • 2
    If you're a Python user, there's the chardet library. This search turned up this answer, and others that may be useful.
    – Seamus
    Apr 24, 2021 at 6:50
  • 1
    Encoding of files is not stored by the file system. You have to examine the data and make a best guess... For example, If you encounter anything that is not valid for an encoding, you can rule out that encoding. (Note, however, that most encodings use the entire range of 0-255 byte values, so it's difficult to do this!) UTF-16 (and some UTF-8 implementations) may output a BOM a the start of the file which can help you make the best guess, and some applications have file formats that specify the encoding. However, those are done by applications, not the OS or file system. (more...)
    – C. M.
    Apr 24, 2021 at 6:59
  • 1
    Because of the difficulties even in ruling out encodings, there's no error-proof way to determine encoding. As noted, file tries to make a best guess, but it's not always right. Same with libraries for python, perl, etc. If the file does not have a mechanism to explicitly declare encoding, you can do what most processors do: Assume ISO-8859-1 until you have reason to believe otherwise. (You can also assume UTF-*, and fall back to ISO-8859-1 if you encounter an invalid UTF-* sequence.)
    – C. M.
    Apr 24, 2021 at 7:05

1 Answer 1

3

Use file. See the man pages for file(1) and magic(5) for details, but here's a few examples:

I've copied a bunch of files of various kinds into a directory:

$ ls -l 
total 389
-rw-r--r-- 1 cas cas 372976 Apr 24 19:09 a.txt
-rw-r--r-- 1 cas cas     14 Apr 24 19:09 b.txt
-rw-r--r-- 1 cas cas  12060 Apr 24 19:09 c.h
-rwxr-xr-x 1 cas cas   5706 Apr 24 19:09 d.sh*
-rwxr-xr-x 1 cas cas    197 Apr 24 19:09 e.pl*
-rw-r--r-- 1 cas cas      6 Apr 24 19:09 f.txt
-rwxr-xr-x 1 cas cas 203072 Apr 24 19:09 g*
-rwxr-xr-x 1 cas cas  79984 Apr 24 19:09 h.c
-rw-r--r-- 1 cas cas   2975 Apr 24 19:09 i.py
-rw-r--r-- 1 cas cas    648 Apr 24 19:09 j.csv

file will make a best guess, using the patterns found in /etc/magic about what they are:

$ file *
a.txt: UTF-8 Unicode (with BOM) text, with very long lines, with CRLF line terminators
b.txt: Little-endian UTF-16 Unicode text, with no line terminators
c.h:   C++ source, ASCII text
d.sh:  Bourne-Again shell script, ASCII text executable
e.pl:  Perl script text executable
f.txt: ASCII text
g:     ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=4bb4d8a0059d50d87638057168576f5ef205efd4, for GNU/Linux 3.2.0, stripped
h.c:   C source, ASCII text, with CRLF line terminators
i.py:  Python script, ASCII text executable
j.csv: CSV text

With these files, it was 100% correct - it identified them flawlessly. Most of the time, that will be the case, but it's not perfect and can get it wrong sometimes.

Note that file does not care what the filename's "extension" (.txt, .py, .c, etc) is, it examines the contents of the file to determine what it is.

It can also tell me what mime-type it thinks they are:

$ file --mime-type *
a.txt: text/plain
b.txt: text/plain
c.h:   text/x-c++
d.sh:  text/x-shellscript
e.pl:  text/x-perl
f.txt: text/plain
g:     application/x-pie-executable
h.c:   text/x-c
i.py:  text/x-script.python
j.csv: application/csv

and what encoding they use:

$ file --mime-encoding *
a.txt: utf-8
b.txt: utf-16le
c.h:   us-ascii
d.sh:  us-ascii
e.pl:  us-ascii
f.txt: us-ascii
g:     binary
h.c:   us-ascii
i.py:  us-ascii
j.csv: us-ascii
2
  • Thanks for this answer, really going to help with my work and future project.
    – ABC
    Apr 26, 2021 at 2:56
  • I meant to post this two days ago, but got called away from the keyboard and forgot that I hadn't clicked on the "Post Your Answer" button.
    – cas
    Apr 26, 2021 at 3:01

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .