The device block size is usually 512 bytes while the filesystem block size is often 4096 bytes. Why are they different? Why are 512B and 4KB good choices for device and filesystem block sizes? What block size would work best for caching disk reads in a userspace library?


The device block size is the block size with what the system is talking with the HDD controllers. If you want to read/write the HDD, it happens so:

  1. Read:

    1. CPU -> HDD controller: "Please send me the data of block 43623626"
    2. HDD controller -> CPU: "Done, here it is: 0xfce2c0deebed..."
  2. Write:

    1. CPU -> HDD controller: "Please write this data to block 3452345: 0xfce2c0deebed..."
    2. HDD controller -> CPU: "done"

Here the block number means the name of the 2354242th, 512-byte block.

Theoretically, it could be possible to use any block size. Most devices are using 512-byte blocks, and some of them, particularly large HDDs are using 4096-byte blocks. Some optical media are using 2304byte blocks.

The important thing is: the block device controller doesn't know anything from the filesystem on it. It can only read and write blocks, in its block size, to his medium. This is what the block device driver uses to provide the block device for the kernel: essentially a single, large byte array. It doesn't matter, how is it partitioned or which fs is using it.

The filesystem block size is the block size in which the filesystem data structures are organized in the filesystem. It is the internal feature of the filesystem, there isn't even a requirement to use block-oriented data structures, and some filesystems doesn't even do it.

Ext4 uses most typically 4096byte blocks.

Furthermore, disk IO data is handled typically not directly by the processes, but with the virtual memory of your OS. It uses extensively paging. The VM page size is typically 4096 bytes (might be different on non-x86 CPUs), it is determined by the CPU architecture. (For example, newer amd64 CPUs can handle 2MB pages, or dec alpha used 8192 byte pages).

To optimize the data IO, the best if all of them are the multiply of eachother, yet better if they are equal. This typically means: use 4096 byte fs blocks.

It is also important: if your block device is partitioned, the partitions should begin/end of exact page sizes. If you don't do it, for example your sda1 starts on the 17. block of your sda, the CPU will have to issue TWO read/write commands for all page read/write operations, because the physical and the filesystem blocks will overlap.

In the most common scenario, it means: all partitions should start or begin on a sector divisible by 8 (4096 / 512 = 8).

Note, typically the low level block IO happens not in single block read/write operations, instead multiple blocks are sent/received in a single command. And re-organizing data is typically not a very big overhead, because memory IO is typically much faster that block device IO. Thus, not following these won't be a big overhead.

  • The exact overhead is dependent on your system, it will be at most some %s and surely below 10%. In a professional scenario, it simply doesn't worth a repartitioning. LVM uses 4MB data blocks, it is multiple of anything, this is yet another reason to use LVM instead of the fossil partitions. – peterh - Reinstate Monica Jan 14 '19 at 7:07
  • All right I like what you said. So I plan to make it 4096 bytes. But I couldn't control how much and where the application wants to read. For example, the application may read across my block boundaries and this misalignment is inherent right? As a cache I can only try to be consistent with the underlying system, but have no control on those sit over me. Is there a huge performance difference between using read/write and directly mmap the block device? – Cyker Jan 14 '19 at 7:24
  • @Cyker It doesn't matter, how your app handles file IO, the fs or the block layer will convert them to full-block operations. File read/write copies-writes data between the block cache and the app memory, while mmap maps directly the pages. Thus, mmap is mostly faster, but not always. With read/write, you have more control, what exactly should be read/written, in mmap the VM heuristics will be used, which are quite good, but won't be ever so good, than an app which is programmed by you, knowing what should be done and why. – peterh - Reinstate Monica Jan 14 '19 at 8:20
  • @Cyker What I see a significant performance improvement: many filesystems and processes have a braindamagedly low write cache sync timeout. For example, ext4 syncs write cache in every 5 seconds!!! Or: postgresql makes an fsync() after every write operation! I can't understand, why need anyways high quality softwares to be so... dirty. It totally shocks me. These all can be turned off, but not these are the default setting! – peterh - Reinstate Monica Jan 14 '19 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.