Finding text file's encoding

Question

The iconv command may change file encodings. But is there a command to find the file encoding of a certain text file. Also, I am trying to find what file encoding standards each country use. So that I may change the encoding to the correct ISO standard.

Which command may find a file encoding? .txt extension. Or others such as .py, or .c, source code files?

Then change it to the correct countries encoding. Based on a countries standard. I am trying to find references for the correct endian formats, and other relative documentation.

Making a text file created UTF-8 in America, formatted for Russian. As if it was made in Russian formatted compatibility.

Answer 1

Use file. See the man pages for file(1) and magic(5) for details, but here's a few examples:

I've copied a bunch of files of various kinds into a directory:

$ ls -l 
total 389
-rw-r--r-- 1 cas cas 372976 Apr 24 19:09 a.txt
-rw-r--r-- 1 cas cas     14 Apr 24 19:09 b.txt
-rw-r--r-- 1 cas cas  12060 Apr 24 19:09 c.h
-rwxr-xr-x 1 cas cas   5706 Apr 24 19:09 d.sh*
-rwxr-xr-x 1 cas cas    197 Apr 24 19:09 e.pl*
-rw-r--r-- 1 cas cas      6 Apr 24 19:09 f.txt
-rwxr-xr-x 1 cas cas 203072 Apr 24 19:09 g*
-rwxr-xr-x 1 cas cas  79984 Apr 24 19:09 h.c
-rw-r--r-- 1 cas cas   2975 Apr 24 19:09 i.py
-rw-r--r-- 1 cas cas    648 Apr 24 19:09 j.csv

file will make a best guess, using the patterns found in /etc/magic about what they are:

$ file *
a.txt: UTF-8 Unicode (with BOM) text, with very long lines, with CRLF line terminators
b.txt: Little-endian UTF-16 Unicode text, with no line terminators
c.h:   C++ source, ASCII text
d.sh:  Bourne-Again shell script, ASCII text executable
e.pl:  Perl script text executable
f.txt: ASCII text
g:     ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=4bb4d8a0059d50d87638057168576f5ef205efd4, for GNU/Linux 3.2.0, stripped
h.c:   C source, ASCII text, with CRLF line terminators
i.py:  Python script, ASCII text executable
j.csv: CSV text

With these files, it was 100% correct - it identified them flawlessly. Most of the time, that will be the case, but it's not perfect and can get it wrong sometimes.

Note that file does not care what the filename's "extension" (.txt, .py, .c, etc) is, it examines the contents of the file to determine what it is.

It can also tell me what mime-type it thinks they are:

$ file --mime-type *
a.txt: text/plain
b.txt: text/plain
c.h:   text/x-c++
d.sh:  text/x-shellscript
e.pl:  text/x-perl
f.txt: text/plain
g:     application/x-pie-executable
h.c:   text/x-c
i.py:  text/x-script.python
j.csv: application/csv

and what encoding they use:

$ file --mime-encoding *
a.txt: utf-8
b.txt: utf-16le
c.h:   us-ascii
d.sh:  us-ascii
e.pl:  us-ascii
f.txt: us-ascii
g:     binary
h.c:   us-ascii
i.py:  us-ascii
j.csv: us-ascii