Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

When I sum up the sizes of my files, I get one figure. If I run du, I get another figure. If I run du on all the files on my partition, it doesn't match what df claims is used. Why are there so many different figures for the total size of my files? Can't computers add?

Speaking of adding: when I add the “Used” and “Available” columns of df, I don't get the total figure. And that total figure is smaller than the size of my partition. And if I add up my partition sizes I don't get my disk size! What gives?

share|improve this question

1 Answer 1

up vote 62 down vote accepted

Adding up numbers is easy. The problem is, there are many different numbers to add.

How much disk space does a file use?

The basic idea is that a file containing n bytes uses n bytes of disk space, plus a bit for some control information: the file's metadata (permissions, timestamps, etc.), and a bit of overhead for the information that the system needs to find where the file is stored. However there are many complications.

Microscopic complications

Think of each file as a series of books in a library. Smaller files make up just one volume, but larger files consist of many volumes, like an encyclopedia. In order to be able to locate the files, there is a card catalog which references every volume. Each volume has a bit of overhead due to the covers. If a file is very small, this overhead is relatively large. Also the card catalog itself takes up some room.

Going a bit more technical, in a typical simple filesystem, the space is divided in blocks. A typical block size is 4kB. Each file takes up an integer number of blocks. Unless the file size is a multiple of the block size, the last block is only partially used. So a 1-byte file and a 4096-byte file both take up 1 block, whereas a 4097-byte file takes up two blocks. You can observe this with the du command: if your filesystem has a 4kB block size, then du will report 4kB for a 1-byte file.

If a file is large, then additional blocks are needed just to store the list of blocks that make up the file (these are indirect blocks; more sophisticated filesystems may optimize this in the form of extents). The output of du doesn't account for this, it only reports the number of blocks used by the file itself.

Some filesystems try to reuse the free space left in the last block to pack several file tails in the same block. Some filesystems (such as ext4 since Linux 3.8 use 0 blocks for tiny files (just a few bytes) that entirely fit in the inode.

Macroscopic complications

Generally, as seen above, the total size reported by du is the file size, rounded up to a number of blocks.

The size reported by du may be smaller if the file is compressed. Unix systems traditionally support a crude form of compression: if a file block contains only null bytes, then instead of storing a block of zeroes, the filesystem can omit that block altogether. A file with omitted blocks like this is called a sparse file. Sparse files are not automatically created when a file contains a large series of null bytes, the application must arrange for the file to become sparse.

Some filesystems such as btrfs and zfs support general-purpose compression.

Advanced complications

Two major features of very modern filesystems such as zfs and btrfs make the relationship between file size and disk usage significantly more distant: snapshots and deduplication.

Snapshots are a frozen state of the filesystem at a certain date. Filesystems that support this feature can contain multiple snapshots taken at different dates. These snapshots take room, of course. At one extreme, if you delete all the files from the active version of the filesystem, the filesystem won't become empty if there are snapshots remaining.

Any file or block that hasn't changed since a snapshot, or between two snapshots was taken exists identically in the snapshot and in the active version or other snapshot. This is implemented via copy-on-write. In some edge cases, it's possible that deleting a file on a full filesystem will fail due to insufficient available space — because removing that file would require making a copy of a block in the directory, and there's no more room for even that one block.

Deduplication is a storage optimization technique that consists of avoiding storing identical blocks. With typical data, looking for duplicates isn't always worth the effort. Both zfs and btrfs support deduplication as an optional feature.

Why is the total from du different from the sum of the file sizes?

As we've seen above, the size reported by du for each file is normally the size of the file, rounded up to the next integer number of blocks. Note that by default, ls -l lists sizes in bytes, but du lists sizes in kB, or in 512-byte units (sectors) on some more traditional systems (du -k or du -P forces the use of kilobytes). Most modern unices support ls -lh and du -h to use “human-readable” numbers using kB, MB, GB, etc. as appropriate.

When you run du on a directory, it sums up the disk usage of all the files in the directory tree, including the directories themselves. A directory contains data (the names of the files, and a pointer to where the file's metadata is), so it needs a bit of storage space. A small directory will take up one block, a larger directory will require more blocks. When you run ls -ld /some/directory, the directory's size is listed. (Note that the “total NNN” line at the top of the output from ls -l is an unrelated number, it's the sum of the sizes in blocks of the listed items, expressed in kB or sectors.)

Keep in mind that du includes dot files which ls doesn't show unless you use the -A or -a option.

Sometimes du reports less than the expected sum. This happens if there are hard links inside the directory tree: du counts each file only once.

Beware that if there are mount points under a directory, du will count all the files on these mount points as well, unless given the -x option. So if for instance you want the total size of the files in your root filesystem, run du -x /, not du /.

Deleted files

When a file is deleted, this only removes the directory entry, not necessarily the file itself. Two conditions are necessary in order to actually delete a file and thus reclaim its disk space:

  • The file's link count must drop to 0: if a file has multiple hard links, removing one doesn't affect the others.
  • As long as the file is open by some process, the data remains. Only when all processes have closed the file is the file deleted. The output fuser -m or lsof on a mount point includes the processes that have a file open on that filesystem, even if the file is deleted.

If you delete a file in some file managers or GUI environments, it may be put into a trash area where it can be undeleted. As long as the file can be undeleted, its space is still consumed.

What are these numbers from df exactly?

A typical filesystem contains:

  • Blocks containing file data (including directories).
  • Free blocks.
  • Blocks that are reserved to the root user.
  • Indirect blocks, superblocks and other control information.
  • Inodes
  • A journal

Only the first kind is reported by du. When it comes to df, what goes into the “used”, “available” and total columns depends on the filesystem (of course used blocks (including indirect ones) are always in the “used” column, and unused blocks are always in the “available” column).

Filesystems in the ext2/ext3/ext4 reserve 5% of the space to the root user. This is useful on the root filesystem, to keep the system going if it fills up (in particular for logging, and to let the system administrator store a bit of data while fixing the problem). Even for data partitions such as /home, keeping that reserved space is useful because an almost-full filesystem is prone to fragmentation. Linux tries to avoid fragmentation (which slows down file access, especially on rotating mechanical devices such as hard disks) by pre-allocating many consecutive blocks when a file is being written, but if there are not many consecutive blocks, that can't work.

Traditional filesystems, up to and including ext4 but not btrfs, reserve a fixed number of inodes when the filesystem is created. This significantly simplifies the design of the filesystem, but has the downside that the number of inodes needs to be sized properly: with too many inodes, space is wasted; with too few inodes, the filesystem may run out of inodes before running out of space. The command df -i reports how many inodes are in use and how many are available (filesystems where the concept is not applicable may report 0).

Running tune2fs -l on the volume containing an ext2/ext3/ext4 filesystem reports some statistics including the total number and number of free inodes and blocks.

Another feature that can confuse matter is subvolumes (supported in btrfs, and in zfs under the name datasets). Multiple subvolumes share the same space, but have separate directory tree roots.

If a filesystem is mounted over the network (NFS, Samba, etc.) and the server exports a portion of that filesystem (e.g. the server has a /home filesystem, and exports /home/bob), then df on a client reflects the data for the whole filesystem, not just for the part that is exported and mounted on the client.

What's using the space on my disk?

As we've seen above, the total size reported by df does not always take all the control data of the filesystem into account. Use filesystem-specific tools to get the exact size of the filesystem if needed. For example, with ext2/ext3/ext4, run tune2fs -l and multiply the block size by the block count.

When you create a filesystem, it normally fills up the available space on the enclosing partition or volume. Sometimes you might end up with a smaller filesystem when you've been moving filesystems around or resizing volumes.

Use volume management or partitioning tools to check what partitions you have: lvs, vgs, pvs for LVM, fdisk for traditional PC-style (“MBR”) partitions, gdisk for GPT partitions, disklabel for BSD disklabels, Parted, etc. Under Linux, cat /proc/partitions gives a quick summary. Typical installations have at least two partitions or volumes used by the operating system: a filesystem (sometimes more), and a swap volume.

Some computers have a partition containing the BIOS or other diagnostic software. Computers with UEFI have a dedicated bootloader partition.

Finally, note that most computer programs use units based on powers of 1024 = 210 (because programmers love binary and powers of 2). So 1 kB = 1024 B, 1 MB = 1048576 B, 1 GB = 1073741824, 1 TB = 1099511627776 B, … Officially, these units are known as kibibyte kiB, mebibyte MiB, etc., but most software just reports k or kB, M or MB, etc. On the other hand, hard disk manufacturers systematically use metric (1000-based units). So that 1 TB drive is only 931 GiB.

share|improve this answer
    
does the use of tune2fs implies to be root ? I tried on a system on which I'm not root and no luck :-( –  Kiwy Mar 19 at 9:37
1  
@Kiwy tune2fs requires having read access to the block device that contains the filesystem, which in general requires being root since that lets you read the content of any file. –  Gilles Mar 19 at 10:10
11  
I know that 'thank you' is discouraged in SE, but Gilles you deserve a huge 'Thank you' for this terrific post. –  dotancohen Mar 19 at 10:52
1  
I remember seeing a card catalog when I was like 6. I wonder how many won't know what they are? –  Izkata Mar 19 at 14:32
1  
@illuminÉ That's too advanced Solaris for me, I don't know at what level it fits. –  Gilles Mar 19 at 15:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.