Linux file command classifying files

Question

I need to recognize type of data contained in random files. I am new to Linux.

I am planning to use the file command to understand what type of data a file has. I tried that command and got the output below.

Someone suggested to me that the file command looks at the initial bytes of a file to determine data type. The file command doesn't look at a file extension at all. Is that correct? I looked at the man page but felt that it was too technical. I would appreciate if anyone can provide a link which has much simpler explanation regarding how the file command works.

What are different possible answers that I could get after running the file command? For example, in the transcript below I get JPEG, ISO media, ASCII, etc:

The screen output is as follows

 m7% file date-file.csv
date-file.csv: ASCII text, with CRLF line terminators
m7% file image-file.JPG
image-file.JPG: JPEG image data, EXIF standard
m7% file music-file.m4a
music-file.m4a: ISO Media, MPEG v4 system, iTunes AAC-LC
m7% file numbers-file.txt
numbers-file.txt: ASCII text
m7% file pdf-file.pdf
pdf-file.pdf: PDF document, version 1.4
m7% file text-file.txt
text-file.txt: ASCII text
m7% file video-file.MOV
video-file.MOV: data

---

Update 1

Thanks for answers and they clarified a couple of things for me.

So if I understand correctly folder /usr/share/mime/magic has a database that will give me what are the current possible file formats (outputs that I can get when I type file command and follow it by a file). is that correct? Is it true that whenever 'File' command output contains the word "text" it refers to something that you can read with a text viewer, and anything without "text" is some kind of binary?

Answer 1

file uses several kinds of test:

1: If file does not exist, cannot be read, or its file status could not be determined, the output shall indicate that the file was processed, but that its type could not be determined.

This will be output like cannot open file: No such file or directory.

2: If the file is not a regular file, its file type shall be identified. The file types directory, FIFO, socket, block special, and character special shall be identified as such. Other implementation-defined file types may also be identified. If file is a symbolic link, by default the link shall be resolved and file shall test the type of file referenced by the symbolic link. (See the -h and -i options below.)

This will be output like .: directory and /dev/sda: block special. Much of the format for this and the previous point is partially defined by POSIX - you can rely on certain strings being in the output.

3: If the length of file is zero, it shall be identified as an empty file.

This is foo: empty.

4: The file utility shall examine an initial segment of file and shall make a guess at identifying its contents based on position-sensitive tests. (The answer is not guaranteed to be correct; see the -d, -M, and -m options below.)

5: The file utility shall examine file and make a guess at identifying its contents based on context-sensitive default system tests. (The answer is not guaranteed to be correct.)

These two use magic number identification and are the most interesting part of the command. A magic number is a special sequence of bytes that's in a known place in a file that identifies its type. Traditionally that place is the first two bytes, but the term has been extended further to include longer strings and other locations. See this other question for more detail about magic numbers in the file command.

The file command has a database of these numbers and what type they correspond to; that database is usually in /usr/share/mime/magic, and maps file contents to MIME types. The output there (often part of file -i if you don't get it by default) will be a defined media type or an extension. "Context-sensitive tests" use the same sort of approach, but are a bit fuzzier. None of these are guaranteed to be right, but they're intended to be good guesses.

file also has a database mapping those types to names, by which it will know that a file it has identified as application/pdf can be described as a PDF document. Those human-readable names may be localised to another language too. These will always be some high-level description of the file type in a way a person will understand, rather than a machine.

The majority of different outputs you can get will come from these stages. You can look at the magic file for a list of supported types and how they're identified - my system knows 376 different types. The names given and the types supported are determined by your system packaging and configuration, and so your system may support more or fewer than mine, but there are generally a lot of them. libmagic also includes additional hard-coded tests in it.

6: The file shall be identified as a data file.

This is foo: data, when it failed to figure out anything at all about the file.

There are also other little tags that can appear. An executable (+x) file will include "executable" in the output, usually comma-separated. The file implementation may also know extra things about some file formats to be able to describe additional points about them, as in your "PDF document, version 1.4".

Answer 2

Man pages are usually terse references, not introductions. Start with the Wikipedia page.

file looks only at the file content, not at the file name. (It also looks at some file metadata such as the file type: directory, symbolic link, named pipe, etc. But in the cases you're interested in, it's the content that matters.)

file typically guesses the format of a file by looking at the first few bytes and comparing them with a built-in table of magic numbers. For example, if the file begins with %PDF, then file reports “PDF document” (and goes digging further to report the minimum version). For file types that don't start with magic numbers, it contains heuristics, e.g. report “ASCII text” if the first few bytes are all in the printable ASCII range.

The output of file is fragile: it can vary from unix variant to unix variant and from version to version. On Linux, Cygwin and *BSD, the file command supports an option -i which produces predictable output in the form of a MIME media type (IANA manages the list of standard media types). There's aren't as many details and the output is less human-friendly but the output is predictable and computer-friendly.

$ file -i somefile.csv
somefile.csv: text/plain; charset=us-ascii
$ file -i somefile.jpg
somefile.jpg: image/jpeg; charset=binary
$ file -i somefile.pdf
somefile.pdf: application/pdf; charset=binary

Use file --mime-type if you only want the MIME type itself without encoding information, e.g. application/pdf. Pass the option -b if you don't want to display the file name at the beginning of the line.

Answer 3

I would like you to read the answer from here. Some of the excerpts from the answer are,

From man page of file command,

file command actually performs 3 tests on determining the file type.

First test

The filesystem tests are based on examining the return from a stat(2) system call.

Second test

The magic number tests are used to check for files with data in particular fixed formats.

Third test

The language tests look for particular strings (cf names.h) that can appear anywhere in the first few blocks of a file. For example, the keyword .br indicates that the file is most likely a troff(1) input file, just as the keyword struct indicates a C program.

The output of the file command is generally based on the result of any of the tests that succeeds.

Now, assuming the C++ program starts like this, and the third test succeeds,

#include <iostream>
bla
bla

As per the third test the keyword #include particularly specifies it is of type C program though we have a CPP program in hand. Now, when I check,

$ file example.cpp

example.cpp: ASCII C program text

Now, the concepts of object oriented are specific to C++. Let us create a file specific to C++.

I start my C++ program as,

class something
{
}
bla
bla

Now, when I issue

$ file example.cpp

The output is,

example.cpp: ASCII C++ program text

This basically explains on how file command works on similar files (In this example, C program and C ++ program are treated alike unless and until we use the object oriented features specific to C++).

Answer 4

Gilles and Michael Homer have provided excellent answers. which I refer you to. To see the types of files recognised on your system, try running

cat /usr/share/magic

If that gives permission issues, or doesn't exist, then possibly

find / -exec file {} \; 2>/dev/null | cut -d":" -f2 | sort -u

(may need tweaking depending on your system) which should show you a list of file types on your system. This command may well take a long time to run depending on the size of your root file system.