I have a very high density virtualized environment with containers, so I'm trying to make each container really small. "Really small" means 87 MB on base Ubuntu 14.04 (Trusty Tahr) without breaking up the package manager compatibility.
So I use LVM as a backing storage for my containers and recently I found very strange numbers. Here they are.
Let's create a 100 MiB (yeah, power of 2) logical volume.
sudo lvcreate -L100M -n test1 /dev/purgatory
I'd like to check the size, so I issue
sudo lvs --units k
test1 purgatory -wi-a---- 102400.00k
Sweet, this is really 100 MiB.
Now let's make an ext4 filesystem. And of course, we remember
-m 0 parameter, which prevents space waste.
sudo mkfs.ext4 -m 0 /dev/purgatory/test1 mke2fs 1.42.9 (4-Feb-2014) Filesystem label= OS type: Linux Block size=1024 (log=0) Fragment size=1024 (log=0) Stride=0 blocks, Stripe width=0 blocks 25688 inodes, 102400 blocks 0 blocks (0.00%) reserved for the super user First data block=1 Maximum filesystem blocks=67371008 13 block groups 8192 blocks per group, 8192 fragments per group 1976 inodes per group Superblock backups stored on blocks: 8193, 24577, 40961, 57345, 73729 Allocating group tables: done Writing inode tables: done Creating journal (4096 blocks): done Writing superblocks and filesystem accounting information: done
Sweet and clean. Mind the block size - our logical volume is small, so mkfs.ext4 decided to make a 1 KiB sized block, not the usual 4 KiB.
Now we will mount it.
sudo mount /dev/purgatory/test1 /mnt/test1
And let's call
df without parameters (we would like to see 1 KiB-blocks)
/dev/mapper/purgatory-test1 95054 1550 91456 2% /mnt/test1
Wait, oh shi~
We have 95054 blocks total. But the device itself has 102400 blocks of 1 KiB. We have only 92.8% of our storage. Where are my blocks, man?
Let's look at it on a real block device. A have a 16 GiB virtual disk, 16777216 blocks of 1K, but only 15396784 blocks are in df output. 91.7%, what is it?
Now follows the investigation (spoiler: no results)
Filesystem could begin not at the beginning of the device. This is strange, but possible. Luckily, ext4 has magic bytes, let's check their presence.
sudo hexdump -C /dev/purgatory/test1 | grep "53 ef"
This shows superblock:
00000430 a9 10 e7 54 01 00 ff ff 53 ef 01 00 01 00 00 00 |...T....S.......|
Hex 430 = Dec 1072, so somewhere after first kilobyte. Looks reasonable, ext4 skips first 1024 bytes for oddities like VBR, etc.
- This is journal!
No, it is not. Journal take space from Available if df output.
- Oh, we have dump2fs and could check the sizes there!
... a lot of greps ...
sudo dumpe2fs /dev/purgatory/test1 | grep "Free blocks"
Free blocks: 93504 Free blocks: 3510-8192 Free blocks: 8451-16384 Free blocks: 16385-24576 Free blocks: 24835-32768 Free blocks: 32769-40960 Free blocks: 41219-49152 Free blocks: 53249-57344 Free blocks: 57603-65536 Free blocks: 65537-73728 Free blocks: 73987-81920 Free blocks: 81921-90112 Free blocks: 90113-98304 Free blocks: 98305-102399
And we have another number. 93504 free blocks.
The question is: what is going on?
- Block device: 102400k (lvs says)
- Filesystem size: 95054k (df says)
- Free blocks: 93504k (dumpe2fs says)
- Available size: 91456k (df says)