55

I would like to know how file types are known if filenames don't have suffixes.

For example, a file named myfile could be binary or text to start with, how does the system know if the file is binary or text?

  • 3
    Just a comment, the rest of the answers cover everything. Nowadays it may happen that with a misconfigured locale or old executables, some utf-8 files may be misdetected as binary data due to non-ascii bytes. – orion Jun 3 '15 at 13:41
  • 19
    The system doesn't care. Certain applications may care, but they each have their own ways of handling this. – jwodder Jun 3 '15 at 14:55
  • 2
    Note that even for regular files (not device files, unix domain sockets, named pipes, etc) "file type" can mean two different things: (1) A particular file format (".docx", XML, MS-DOS text format, RTF, fixed-length records, the list could be very long) or (2) A file that a particular app knows how to deal with (".xlsx" or ".doc" or whatever, there's overlap with the format type). It's worth keeping that distinction in mind when talking about "file type". – Bruce Ediger Jun 3 '15 at 15:03
  • @jwodder The system does care. It's the system that complains you can't execute a non-executable file when you try to, not those applications! – Mr Lister Jun 4 '15 at 17:36
  • 1
    @MrLister True, but executable/non-executable has nothing to do with 'extension'. – user2338816 Jun 6 '15 at 10:49
83
+50

The file utility determines the filetype over 3 ways:

First the filesystem tests: Within those tests one of the stat family system calls is invoked on the file. This returns the different unix file types: regular file, directory, link, character device, block device, named pipe or a socket. Depending on that, the magic tests are made.

The magic tests are a bit more complex. File types are guessed by a database of patterns called the magic file. Some file types can be determined by reading a bit or number in a particular place within the file (binaries for example). The magic file contains "magic numbers" to test the file whether it contains them or not and which text info should be printed. Those "magic numbers" can be 1-4Byte values, strings, dates or even regular expressions. With further tests additional information can be found. In case of an executable, additional information would be whether it's dynamically linked or not, stripped or not or the architecture. Sometimes multiple tests must pass before the file type can be truly identified. But anyway, it doesn't matter how many tests are performed, it's always just a good guess.

Here are the first 8 bytes in a file of some common filetypes which can help us to get a feeling of what these magic numbers can look like:

             Hexadecimal          ASCII
PNG   89 50 4E 47|0D 0A 1A 0A   ‰PNG|....
JPG   FF D8 FF E1|1D 16 45 78   ÿØÿá|..Ex
JPG   FF D8 FF E0|00 10 4A 46   ÿØÿà|..JF
ZIP   50 4B 03 04|0A 00 00 00   PK..|....
PDF   25 50 44 46|2D 31 2E 35   %PDF|-1.5

If the file type can't be found over magic tests, the file seems to be a text file and file looks for the encoding of the contents. The encoding is distinguished by the different ranges and sequences of bytes that constitute printable text in each set.

The line breaks are also investigated, depending on their HEX values:

  • 0A (\n) classifies a Un*x/Linux/BSD/OSX terminated file
  • 0D 0A (\r\n) are file from Microsoft operating systems
  • 0D (\r) would be Mac OS until version 9
  • 15 (\025) would be IBMs AIX

Now the language tests start. If it appears to be a text file, the file is searched for particular strings to find out which language it contains (C, Perl, Bash). Some script languages can also be identified over the hashbang (#!/bin/interpreter) in the first line of the script.

If nothing applies to the file, the file type can't be determined and file just prints "data". This is also the case if the file is encrypted.

So, you see there is no need for a suffix. A suffix anyway could confuse, if set wrong.

  • 4
    There's also the freedesktop.org shared MIME database, which is used by essentially all X11 applications. This is similar in concept to what file(1) does, but with a (very) different implementation. – lcd047 Jun 3 '15 at 16:34
  • 4
    Note that the result of this process is basically a guess, and shouldn't be relied upon for anything important. (Convenience features, like deciding the default program to open the file with, are fine) – immibis Jun 4 '15 at 0:10
  • So if I add %PNG at the top of a text file, it will be seen as a png file. Right?? – saga Jan 15 '17 at 5:26
19

Often, it doesn't care. You just pass it to a program and either it interprets it or it doesn't. It may not be useful to open a .jpg in a text editor, but you're not prevented from doing this. The extension, like the rest of the filename, is for the organisational convenience of humans.

It may also be possible to construct files that can be validly interpreted in multiple ways. Because the ZIP file format starts has a header at the end of the file, you can prepend other things to the front and it will still load as a ZIP file. This is commonly used to make self-extracting zip files.

  • 3
    Re the last paragraph: Funky File Formats is an interesting talk on that subject, presenting e.g. a jpeg that is also a java hello world program, after AES encrypting it it becomes a PNG, or after 3DES decrypting it it becomes a PDF and more (all with "interesting" content, i.e. not just with white noise or artefacts) – Hagen von Eitzen Jun 4 '15 at 16:23
14

That information is commonly found in the header of the file. The file command analyzes the target and tells you information about the file. A lot of information is often derived from file headers which are often times the first few bytes of a file (see below). Headers are used by the system to figure out how to handle files. #!/bin/bash at the beginning of a file tells the system to use the bash shell to interpret the following script. ELF tells the system that this is an ELF executable.

[~] root@www # file /bin/ls
/bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped

[~] root@www # file /etc/passwd
/etc/passwd: ASCII text

File header examples:

[root@server4 ~]# xxd old_sm_logo.png | head -5
0000000: 8950 4e47 0d0a 1a0a 0000 000d 4948 4452  .PNG........IHDR
0000010: 0000 0134 0000 006f 0806 0000 0062 bf3c  ...4...o.....b.<

[root@server4 ~]# xxd /bin/ls | head -5
0000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
0000010: 0200 3e00 0100 0000 a024 4000 0000 0000  ..>......$@.....

[root@server4 proj]# xxd resizer.sh | head -5
0000000: 2321 2f62 696e 2f62 6173 680a 5b20 2d7a  #!/bin/bash.[ -z
0000010: 2022 2431 2220 5d20 2626 2065 6368 6f20   "$1" ] && echo
  • 3
    This is rather misleading. Unix files don't have a "header" per se. The file command tries to guess from the contents of the file how the file is probably intended to be used. It is not infallible. – Nate Eldredge Jun 3 '15 at 21:11
  • You are right in how you explained the behavior of file. It does in fact make an analysis of the file. However, most file types are identified by a header of sorts. 0000000: 7f45 4c46 0201 0100 0000 0000 0000 0000 .ELF............ is a header of an ELF executable (first few bytes of /bin/ls). Similarly #!/bin/bash at the top of an ASCII file would identify it as a shell script. Another example: 0000000: 8950 4e47 0d0a 1a0a 0000 000d 4948 4452 .PNG........IHDR (a .png image) – h3rrmiller Jun 3 '15 at 21:34
  • 2
    But your answer makes it sound like a header is an inherent feature of a Unix file. Text files, for instance, have no such header; someone like the OP would probably consider a C source file and a Java source file to have different "file types", but there is no header to distinguish them. I would argue that "file type" is not even a meaningful concept under Unix; the operating system just provides a file system, and it is up to each application to decide what the contents of any given file mean. – Nate Eldredge Jun 3 '15 at 22:38
  • I agree. I was trying to answer as simply as possible without going down too many rabbit holes. – h3rrmiller Jun 3 '15 at 22:45
7

The first thing to check is the hard-coded file type that is recognized by the kernel. These are the file types such as directory, character-special file, block-special file, pipe-special file, socket, and symbolic link. This information comes from the inode of the file. If the file is a plain file, the next set of information comes from the first 256 bytes by looking for patterns. Thus, the text files and C source code are recognized by examining those bytes. In addition, the utilities also look for a magic number that is used to test and validate the file type. You can add your own file types to be recognized by adding the information to the file /etc/magic. Refer to the man page for magic(5) to see the format of the magic file.

In older implementation (Solaris, for example), the file /etc/magic enumerated most of the file types recognized.

4

The file command applies some heuristics from inspecting (parts of) the file and making a qualified guess. Beyond that there are some special cases where additional information can be obtained; like the #! at the beginning of a text file, a BoM (byte order mark), or specific header bytes of executable file formats. The #! and binary marks in executables are used by the system to tell them apart.

4

The system doesn't know whether a file is binary or text. In all (AFAIK) Unix-type operating systems, fopen(path, "rb") is exactly the same as fopen(path "r") - the b has no effect. It's accepted because standard C needs to be portable to some other OSes that do make such a distinction.

0

I would argue that "file type" is not even a meaningful concept under Unix;

In good old days of mainframe commputers their OS's supported several file types including sequential and index-sequential. Modern operating systems (Un*x and arguably Windows) reduce the set of the file types to a minimum (including executable, shared object).

It may also be possible to construct files that can be validly interpreted in multiple ways

It is possible, there is a tricky file format: a piece of C code which can be interpreted as an image description. Besides there are different formats less and more specific: text file, XML file, a SOAP document.

  • 1
    As file formats go, XPM isn't that tricky. I consider "tricky" to start with something that's both a valid JPEG and a valid ZIP file. – Mark Jun 6 '15 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.