Uniformity of Linux files

Question

How is it that, if you are in a folder with various files --- pictures, binary executables, scripts, even directories, zip files, just about everything --- how it is that when you hit ls -l, you get output that is similar (to the form) for all files?

It is like the files are themselves placed in super files that are uniform, and has meta data attached, sort of like the control block structure of processes.

And how come files that you expect would have nothing to do with Unix or Linux pose no problem? Is there like a file system preprocessor to deal with files from the Windows world? Or are files that cross-platform that it is not necessary? (That would be something!)

Is there a way you could see this hidden (?) layer? (Even HTML (not to mention Lisp etc.) is difficult to understand just reading about it. But when you see it, you understand immediately.)

Answer 1

You've hit on almost exactly the correct answers. You just need some details filled in.

For the most part (special files excluded) every file is treated exactly the same, and directories (a.k.a. "folders") can actually be implemented as files. The files are actually just a series of disk blocks. The "metadata" you write of is called "inodes" (see What is a Superblock, Inode, Dentry, File, and What is an inode on this site). Directories (or "folders") are just a way to match up a file name to an inode The metadata apart from the name (file type, permissions, ownerships, etc) live in the inode. That metadata is in binary form, mostly for compactness and speed of access: parsing XML or indeed any other text format, would take way too long.

You can see a text representation of the file's metadata with the stat command: stat /bin/cat will interest you. stat is a GNU command, so it won't be universally available, but most Linux distros should have it.

As far as "foreign" files, the file/inode/directory concept comes from a time when the variability of operating systsms was far, far greater than it is today. The designers of Unix and its original file system had a lot of experience with what today would be considered very bizarre operating systems and file systems. The file/inode/directory concept almost certainly embodies all their experiences, and so the mild variations in "what is a file" that today's more homogenous OSes provide isn't really a problem. For a take on this from 20 years ago, see The Filesystem Belongs in the Kernel. For a bit more recent take, see The Hideoous Name. That's a Russ Cox blog post, read the Rob Pike paper the blog post references, as well. You will receive enlightenment from it.

Answer 2

All files are a sequence of bytes under the hood. It doesn't matter what is in them. The system only needs to know how many bytes there are in the file, and to arrange for a place to store these bytes. Whatever the contents of the file are, it's a bunch of bytes.

For each file, the system also stores some metadata. The metadata is independent of the content of the file; in fact, you could say that the content is one of these metadata items (but it's usually considered separately, because the content has a variable size and can potentially be very large). What metadata is stored depends on the filesystem type. Unix filesystems typically provide what you see in ls: permissions, a modification time, and so on. The file's name is also a piece of metadata with a somewhat special status: it is unique, i.e. there cannot be two files with the same name (in the same directory).

Files are organized in directories. A directory can contain regular files, subdirectories and some special file types such as symbolic links. There is a directory called the root directory which all directories are subdirectories of n times removed.

The path to a file indicates how to find it starting from the root directory. Each directory to be traversed is given in turn, going from the root downwards, with slash / characters after each directory name. For example, /usr/bin/env means to start at the root directory /, go to the usr subdirectory there, go to the bin subdirectory there, and look up the entry env in that subsubdirectory.

When you list the content of a directory, the kernel (which includes the filesystem drivers) does the job of reading the bytes that represent the directory on the disk, and converting that into a list of file names and metadata and content location. The ls program calls a function called opendir to open the directory, then makes repeated calls to readdir to read each entry in the directory in turn, i.e. to obtain the list of file names in that directory (including the subdirectories and other special files). With ls -l, the ls program needs more information about the file than just the name, so it also calls stat to retrieve each file's metadata.

Some operating systems let you view the on-disk representation of a directory with any text viewing program such as cat. Linux doesn't let you though. You can use the debugfs program to explore the filesystem (use it only in read-only mode!).