How are file types known if not from file suffix?

Question

I would like to know how file types are known if filenames don't have suffixes.

For example, a file named myfile could be binary or text to start with, how does the system know if the file is binary or text?

Answer 1

The file utility determines the filetype over 3 ways:

First the filesystem tests: Within those tests one of the stat family system calls is invoked on the file. This returns the different unix file types: regular file, directory, link, character device, block device, named pipe or a socket. Depending on that, the magic tests are made.

The magic tests are a bit more complex. File types are guessed by a database of patterns called the magic file. Some file types can be determined by reading a bit or number in a particular place within the file (binaries for example). The magic file contains "magic numbers" to test the file whether it contains them or not and which text info should be printed. Those "magic numbers" can be 1-4Byte values, strings, dates or even regular expressions. With further tests additional information can be found. In case of an executable, additional information would be whether it's dynamically linked or not, stripped or not or the architecture. Sometimes multiple tests must pass before the file type can be truly identified. But anyway, it doesn't matter how many tests are performed, it's always just a good guess.

Here are the first 8 bytes in a file of some common filetypes which can help us to get a feeling of what these magic numbers can look like:

             Hexadecimal          ASCII
PNG   89 50 4E 47|0D 0A 1A 0A   ‰PNG|....
JPG   FF D8 FF E1|1D 16 45 78   ÿØÿá|..Ex
JPG   FF D8 FF E0|00 10 4A 46   ÿØÿà|..JF
ZIP   50 4B 03 04|0A 00 00 00   PK..|....
PDF   25 50 44 46|2D 31 2E 35   %PDF|-1.5

If the file type can't be found over magic tests, the file seems to be a text file and file looks for the encoding of the contents. The encoding is distinguished by the different ranges and sequences of bytes that constitute printable text in each set.

The line breaks are also investigated, depending on their HEX values:

• 0A (\n) classifies a Un*x/Linux/BSD/OSX terminated file
• 0D 0A (\r\n) are file from Microsoft operating systems
• 0D (\r) would be Mac OS until version 9
• 15 (\025) would be IBMs AIX

Now the language tests start. If it appears to be a text file, the file is searched for particular strings to find out which language it contains (C, Perl, Bash). Some script languages can also be identified over the hashbang (#!/bin/interpreter) in the first line of the script.

If nothing applies to the file, the file type can't be determined and file just prints "data".

So, you see there is no need for a suffix. A suffix anyway could confuse, if set wrong.

Answer 2

Often, it doesn't care. You just pass it to a program and either it interprets it or it doesn't. It may not be useful to open a .jpg in a text editor, but you're not prevented from doing this. The extension, like the rest of the filename, is for the organisational convenience of humans.

It may also be possible to construct files that can be validly interpreted in multiple ways. Because the ZIP file format starts has a header at the end of the file, you can prepend other things to the front and it will still load as a ZIP file. This is commonly used to make self-extracting zip files.

Answer 3

That information is commonly found in the header of the file. The file command analyzes the target and tells you information about the file. A lot of information is often derived from file headers which are often times the first few bytes of a file (see below). Headers are used by the system to figure out how to handle files. #!/bin/bash at the beginning of a file tells the system to use the bash shell to interpret the following script. ELF tells the system that this is an ELF executable.

[~] root@www # file /bin/ls
/bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped

[~] root@www # file /etc/passwd
/etc/passwd: ASCII text

File header examples:

[root@server4 ~]# xxd old_sm_logo.png | head -5
0000000: 8950 4e47 0d0a 1a0a 0000 000d 4948 4452  .PNG........IHDR
0000010: 0000 0134 0000 006f 0806 0000 0062 bf3c  ...4...o.....b.<

[root@server4 ~]# xxd /bin/ls | head -5
0000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
0000010: 0200 3e00 0100 0000 a024 4000 0000 0000  ..>......$@.....

[root@server4 proj]# xxd resizer.sh | head -5
0000000: 2321 2f62 696e 2f62 6173 680a 5b20 2d7a  #!/bin/bash.[ -z
0000010: 2022 2431 2220 5d20 2626 2065 6368 6f20   "$1" ] && echo

Answer 4

The first thing to check is the hard-coded file type that is recognized by the kernel. These are the file types such as directory, character-special file, block-special file, pipe-special file, socket, and symbolic link. This information comes from the inode of the file. If the file is a plain file, the next set of information comes from the first 256 bytes by looking for patterns. Thus, the text files and C source code are recognized by examining those bytes. In addition, the utilities also look for a magic number that is used to test and validate the file type. You can add your own file types to be recognized by adding the information to the file /etc/magic. Refer to the man page for magic(5) to see the format of the magic file.

In older implementation (Solaris, for example), the file /etc/magic enumerated most of the file types recognized.

Answer 5

The file command applies some heuristics from inspecting (parts of) the file and making a qualified guess. Beyond that there are some special cases where additional information can be obtained; like the #! at the beginning of a text file, a BoM (byte order mark), or specific header bytes of executable file formats. The #! and binary marks in executables are used by the system to tell them apart.

Answer 6

The system doesn't know whether a file is binary or text. In all (AFAIK) Unix-type operating systems, fopen(path, "rb") is exactly the same as fopen(path "r") - the b has no effect. It's accepted because standard C needs to be portable to some other OSes that do make such a distinction.

Answer 7

I would argue that "file type" is not even a meaningful concept under Unix;

In good old days of mainframe commputers their OS's supported several file types including sequential and index-sequential. Modern operating systems (Un*x and arguably Windows) reduce the set of the file types to a minimum (including executable, shared object).

It may also be possible to construct files that can be validly interpreted in multiple ways

It is possible, there is a tricky file format: a piece of C code which can be interpreted as an image description. Besides there are different formats less and more specific: text file, XML file, a SOAP document.

Answer 8

Thanks to chaos' answer above, and here is a one-line script adding suffix to image files missing file extensions using file, just in case anyone needs:

for f in ./*; do suffix=$(cat $f | file - | awk '{print $2}'); mv "$f" "$f.$suffix"; done