Why size reporting for directories is different than other files?

Question

I was wondering why an empty directory occupied 4096 bytes of space and I have seen this question. It is stated that space is allocated in blocks and hence, the size of a new directory is 4096 bytes.

However I am pretty sure that allocation for "normal" files are done in blocks as well. At least it is like that in Windows filesystems and I am guessing that it must be at least similar in ext*.

Now as far as I understood, size listing for other types of files, such as files, symbolic links etc. are done in terms of real size. Because when I create an empty file, I see a 0 as the size. When a type a few characters, I see the < number of characters > bytes as the size etc.

So my question is, although the allocation for other files are done in blocks too, why the policy for reporting the size of a directory and a file differs?

Clarification

I thought the question was clear enough but apparently is wasn't. I will try to clarify the question here.

1) What I think a directory is:

I will try to explain what I think a directory is by the following example. After reading, if it is wrong, please notify me.

Let's say that we have a directory named mydir. And let's say that it contains 3 files, which are: f0, f1 and f2. Let's assume that each file is 1 byte long.

Now, what is mydir? It is a pointer to an inode which contains the following: String "f0" and the inode number which f0 points to. String "f1" and the inode number which f1 points to. And string "f2" and the inode number which f2 points to. (At least this is what I think a directory is. Please correct me if I am wrong.)

Now there may be two methods for calculating the size of a directory:

1) Calculating the size of the inode which mydir points to.

2) Summing the sizes of the inodes which contents of mydir points to.

Although 1 is more counter intuitive, let's assume that it is the method that is being used. (For this question, which method is the method that is actually being used does not matter.) Then, the size of mydir is calculated as the following:

2 + 2 + 2 + 3 * <space_required_to_store_an_inode_number>

2's are because each filename is 2 bytes long.

2) The question:

Now the question: Assuming what I think a directory is correct, the reported size for mydir should be much much less than 4096, no matter method 1 or method 2 is being used to calculate its size.

Now, you will say that the reason it is reported 4096 bytes is because the allocation is done in blocks. Hence, the reported size that big.

But then I will say: Allocation is done in blocks for regular files as well. (See thrig's answer for reference) But nevertheless, their sizes are reported in real sizes. (1 byte if they contain 1 character, 2 bytes if they contain 2 characters etc.)

So my question is, why is the policy for reporting sizes of directories is such different than reporting sizes of regular files?

More clarification:

We know that the initial number of blocks allocated for a non-empty file and for an empty directory is both 8 blocks. (See thrig's answer) So even though allocation is made in the same number of blocks for both regular files and directories, why the reported size for a directory is much bigger?

Answer 1

I think the reason you're confused is because you don't know what a directory is. To do this lets take a step back and examine how Unix filesystems work.

The Unix filesystem has several separate notions for addressing data on disk:

• data blocks are a group of blocks on a disk which have the contents of a file.
• inodes are special blocks on a filesystem, with a numerical address unique within that filesystem, which contains metadata about a file such as:
  • permissions
  • access / modification times
  • size
  • pointers to the data blocks (could be a list of blocks, extents, etc)
• filenames are hierarchical locations on a filesystem root that are mapped to inodes.

In other words, a "file" is actually composed of three different things:

1. a PATH in the filesystem
2. an inode with metadata
3. data blocks pointed to by the inode

Most of the time, users imagine a file to be synonymous to "the entity associated with the filename" - it's only when you're dealing with low-level entities or the file/socket API that you think of inodes or data blocks. Directories are one of those low-level entities.

You might think that a directory is a file that contains a bunch of other files. That's only half-correct. A directory is a file that maps filenames to inode numbers. It doesn't "contain" files, but pointers to filenames. Think of it like a text file that contains entries like this:

• . - inode 1234
• .. - inode 200
• Documents - inode 2008
• README.txt - inode 2009

The entries above are called directory entries. They are basically mappings from filenames to inode numbers. A directory is a special file that contains directory entries.

That's a simplification of course, but it explains the basic idea and other directory weirdness.

• Why don't directories know their own size?
  • Because they only contain pointers to other stuff, you have to iterate over their contents to find the size
• Why aren't directories ever empty?
  • Because they contain at least the . and .. entries. Thus, a proper directory will be at least as small as the smallest filesize that can contain those entries. In most filesystems, 4096 bytes is the smallest.
• Why is it that you need write permission on the parent directory when renaming a file?
  • Because you're not just changing the file, you're changing the directory entry pointing to the file.
• Why does ls show a weird number of "links" to a directory?
  • a directory can be referenced (linked to) by itself, its parent, its children.
• What does a hard link do and how does it differ from a symlink?
  • a hard link adds a directory entry pointing to the same inode number. Because it points to an inode number, it can only point to files in the same filesystem (inodes are local to a filesystem)
  • a symlink adds a new inode which points to a separate filename. Because it refers to a filename it can point to arbitrary files in the tree.

---

But wait! Weird things are happening!

ls -ld somedirectory always shows the filesize to be 4096, whereas ls -l somefile shows the actual size of a file. Why?

Point of confusion 1: when we say "size" we can be referring to two things:

• filesize, which is a number stored in the inode; and
• allocated size, which is the number of blocks associated with the inode times the size of each block.

In general, these are not the same number. Try running stat on a regular file and you'll see this difference.

When a filesystem creates a non-empty file, it usually eagerly allocates data blocks in groups. This is because files have a tendency to grow and shrink arbitrarily fast. If the filesystem only allocated as many data blocks as needed to represent the file, growing / shrinking would be slower, and fragmentation would be a serious concern. So in practice, filesystems don't have to keep reallocating space for small changes. This means that there may be a lot of space on disk that is "claimed" by files but completely unused.

What does the filesystem do with all this unused space? Nothing. Until it feels like it needs to. If your filesystem optimizer tool - maybe an online optimizer running in the background, maybe part of your fsck, maybe built-in to your filesystem itself - feels like it, it may reassign the data blocks of your files - moving used blocks, freeing unused blocks, etc.

So now we come to the difference between regular files and directories: because directories form the "backbone" of your filesystem, you expect that they may need to be accessed or modified frequently and should thus be optimized. And so you don't want them fragmented at all. When directories are created, they always max out all their data blocks in size, even when they only have so many directory entries. This is okay for directories, because, unlike files, directories are typically limited in size and growth rate.

The 4096 reported size of directories is the "filesize" number stored in the directory inode, not the number of entries in the directory. It isn't a fixed number - it's the maximum bytes that will fit into the allocated number of blocks for the directory. Typically, this is 512 bytes/block times 8 blocks allocated for a file with any contents - incidentally, for directories, the filesize and the allocated size are the same. Because it's allocated as a single group, the filesystem optimizer won't move its blocks around.

As the directory grows, more data blocks are assigned to it, and it will also max out those blocks by adjusting the filesize accordingly.

And so ls and stat will show the filesize field of the directory's inode, which is set to the size of the data blocks assigned to it.

Answer 2

I think that the initial, empty, directory size depends on the filesystem. On ext3 and ext4 filesystems I have access to, I also get 4096-byte empty directories. On an NFS-mounted NAS of some sort, I get an 80-byte empty directory. I don't have access to a ReiserFS filesystem, the newly-created, empty directory size there would be interesting.

Traditionally, a directory was a file with a bit set in its inode (the on-disk structure describing the file) that indicated it was a directory. That file was filled with variable-length records. Here's what /usr/include/linux/dirent.h says:

struct dirent64 {
    __u64       d_ino;
    __s64       d_off;
    unsigned short  d_reclen;
    unsigned char   d_type;
    char        d_name[256];
};

You could skip through the directory-file-entries by using the d_off values. If an entry got removed (unlink() system call, used by rm command), the d_off value of the previous entry got increased to account for the missing record. Nothing did any "compacting" of records. It was probably just simplest to show the allocation in terms of the number of bytes in the disk blocks allocated to the file, rather than try to figure out how many bytes in a directory file account for all of the entries, or just up to the last entry.

These days, directories have internal formats like B-trees or Hash Trees. I'm guessing that it's either a big performance improvement to do directories by blocks, or there's "blank space" inside them similar to old school directories, so it's hard to decide what the "real size" in bytes of a directory is, particularly one that's been in use for a while and had files deleted and added to it a lot. Easier just to show number-of-blocks multiplied by bytes-per-block.

Answer 3

A file may have no blocks allocated to it; the -s flag to ls will show this difference, while a directory will have some number of minimum blocks allocated, hence the default size. (Unless you're on some fancy modern filesystem that throws these notions out the window.) For example:

% mkdir testfoo
% cd testfoo/
% mkdir foodir
% touch foofile
% ln -s foofile foosln
% ls -ld foo*
drwxrwxr-x  2 jmates  jmates  512 Oct  5 19:48 foodir
-rw-rw-r--  1 jmates  jmates    0 Oct  5 19:48 foofile
lrwxrwxr-x  1 jmates  jmates    7 Oct  5 19:48 foosln -> foofile
% ls -lds foo*
8 drwxrwxr-x  2 jmates  jmates  512 Oct  5 19:48 foodir
0 -rw-rw-r--  1 jmates  jmates    0 Oct  5 19:48 foofile
0 lrwxrwxr-x  1 jmates  jmates    7 Oct  5 19:48 foosln -> foofile
% 

Note that the symlink here takes no blocks, despite dedicating seven bytes for the details necessary to readlink(2), how curious! Anyways, let's now pad foofile with a byte or two:

% echo >> foofile a
% ls -lds foo*
8 drwxrwxr-x  2 jmates  jmates  512 Oct  5 19:48 foodir
8 -rw-rw-r--  1 jmates  jmates    2 Oct  5 19:49 foofile
0 lrwxrwxr-x  1 jmates  jmates    7 Oct  5 19:48 foosln -> foofile
%

And one can see that the allocated blocks for foofile has jumped to 8 despite there being only two bytes (the a and the newline echo tacked on).

Files can also be sparse, which is another way the reported file size versus actual contents can differ, depending on how the tool interacting with the file handles that sparseness.

Also, the size of the directory can be increased, create many files with very long names and check what happens to the size of the directory (and to the blocks allocated) after each new long filename is created with ls -lds .