0

Assuming that all filesystems are specific, how tools like ls, touch, cat, etc interacting with them? I suppose that 'ls' doesn't know specifics about btrfs for example, but it reads directory entries anyway. When some process writes a file, it doesn't to know about block allocator's specifics for particular filesystem, but anyway, file is successfully writen.
Is there some kind of kernel API that hides underlying filesystem implementation from the processes? It's not too complex to design a custom simple filesystem for academic purposes, and write a programs for manipulating it (mkfs, fsck, etc), but how to tell kernel about filesystem implementation, so other processes can use it?

EDIT:
I understand syscalls in user space, but what I am really interested in is what's happening after that, in kernel space.

1

On any POSIX system, the interface between applications and the kernel is a few function calls: open, read, write, close, etc. An application such as cat calls those functions; it doesn't care how the functions are implemented under the hood.

On Unix systems, those functions are actually system calls: the application calls the kernel. Inside the kernel, a typical architecture is to have a VFS layer, which handles tasks that are independent of the filesystem format (such as locating the proper filesystem, permissions, locking, etc.). Once it has determined which filesystem the file is located on, the VFS layer passes the operation on to the proper filesystem-specific driver.

You can observe the interface between applications and the kernel with a tool like strace on Linux or its equivalent on other Unix platforms (trace, truss, …). Example (omitting the part of the trace corresponding to the startup and the final cleanup of cat):

$ strace cat foo
…
open("foo", O_RDONLY)                   = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
mmap(NULL, 139264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8b9ea14000
read(3, "hello\n", 131072)              = 6
write(1, "hello\n", 6hello
)                  = 6
read(3, "", 131072)                     = 0
munmap(0x7f8b9ea14000, 139264)          = 0
close(3)                                = 0
…

You can see cat calling open, read and close on the file. It also calls fstat and fadvise64, I think only for performance optimizations.

The interface between the VFS and the filesystem driver can't be spied on so easily.

Programs like mkfs and fsck don't go through the kernel's filesystem interface, because they don't work on files, they work on storage areas. They access the block device containing the filesystem directly.

If you want to add support for a new filesystem, you need to write a driver for it. There are two ways to go about it.

  • You can write a driver that runs in the kernel; this gets you the best performance, and it's the only way to implement some feature (e.g. fine-grained access control). But it's also harder to debug (if your driver has a bug, you'll probably need to reboot; if you're lucky you may be able to view an error trace and perhaps even delay the reboot until you've saved your data — better do that in a virtual machine). Look up the documentation of your Unix variant's kernel to see what interface you need to implement.
  • Alternatively, you can use FUSE, which is a filesystem driver that forwards all requests back out of the kernel, so each filesystem driver is implemented as a process. If the filesystem is buggy, just kill the filesystem driver process and the rest of the OS survives. To learn how to write a FUSE filesystem, see the examples and read tutorials such as Sumit Singh's.
  • This is very good answer, and it's candidate for the best, for now. – Boban P. Apr 11 '16 at 8:00
2

The Unix/Linux system offers the POSIX system calls open(2)/close(2)/read(2)/write(2) and stat(2) and some higher-level functions like opendir(3)/closedir(3)/readdir(3), which are enough to write the tools stated (it is easier using the C wrappers). Part of the hard job of the kernel is precisely to make them work on the various filesystems offered, and make the whole work on whatever device they might be residing.

  • That part of the job you mentioned is what I am interested, not regular syscalls from userspace. – Boban P. Apr 11 '16 at 7:54
1

For most tools, the underlying layer is the C Standard Library ("libc"). libc provides a number of low-level file handling routines, such as open, read, and write. These routines in turn interface to the filesystem layer in the kernel, which sits on top of the kernel's block device layer, the device drivers, and finally the hardware.

One implementation of libc is the GNU C Library ("glibc"). You might want to take a look at the official documentation, which describes the low-level file functions of glibc in more detail.

  • Call fd = open('/etc/fstab',O_RDONLY) should open file /etc/fstab, but how it finds data and meta blocks of that file? How libc calls know which filesystem is there and how to interpret bytes on media? – Boban P. Apr 10 '16 at 22:07
  • @BobanP. It doesn't. libc translates the open call (or other low-level file functions) into system calls to the kernel. From that moment on, it's entirely the kernel's job to deal with a file's location/filesystem, block format etc. Read also more about syscalls, the linux kernel, and an overview of linux syscalls. – Guido Apr 10 '16 at 22:36
  • 'From that moment on, it's entirely the kernel's job to deal with a file's location/filesystem, block format etc.' ... and that's exactly what I want to know, not about syscalls. – Boban P. Apr 11 '16 at 7:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.