5

I was under the impression that tar does not compress files. Imagine my surprise when I tar up a million small files (du -h ~ 4.2G) and the resulting tar was a quarter of the size (ls -lh mytar.tar ~ 1.3G)!

Clearly these tiny files were taking up space beyond their reported size, and an answer to another question suggests that each non-empty file takes up at least 1KB regardless of it's size. But where does this 1KB come from, does it differ across filesystems (this is ext4), and does a 1.01 KB file take up 2KB?

In short, how do I measure the true file size, especially many files in a directory? I tried du --apparent-size -h and I'm only getting 437M so I quite confused at the three vastly different numbers.

1
  • 1
    Tar files align files to 512 byte blocks. I believe a lot of file systems (depending on configuration) align to 4096 bytes. If a file does not fill a whole block, it will still consume that block and make it unavailable for other files. So padding in a tar can be as little as 1/8 the size of padding on many file systems. Nov 12 '19 at 15:17
5

As Christopher points out this question is very similar to Why is a text file taking up at least 4kB even when there's just one byte of text in it?

I'm not sure if I personally class it as a duplicate or not.

But where does this 1KB come from

This is more commonly 4KB

File systems are allocated in blocks of bytes (AKA allocation units) not individual bytes. So to store a single byte in a file, that file will need an entire bock. This means the rest of the block is left blank, but no other file can use it.

The origin of this number is unclear, but there's a number of things it fits with. For example at a low level, it's not possible to write single bytes to disk, you can only write a block of them. Modern HDs and even SSDs often have a 4KB limit. Meaning that if you want to write one byte, you must first load 4KB, change that 1 byte and write the entire block back. If you attempt to write a whole block, there is no need to read it's original contents. So file systems aligned to hardware limits are much more efficient.

As Stephen Kitt points out, 4KB is the maximum block size supported for ext3 by many kernels. (Also discussed here). In general larger block sizes have more efficient access times meaning "larger blocks are better".

does it differ across filesystems (this is ext4)

Once apon a time 512 was a common block size, and this figure still comes up occasionally as a default value. Tar files are very old and have this same 512 byte block size (presumably in an attempt to align with the file systems and hardware making disk writes very fast). As such tar files are still very wasteful with very small files (<512 bytes)

It's much more common now to have file systems that are 4KB aligned (not 1KB).

And yes, file systems can be configured when you format them to use a different block size. Different file systems have different limits but most can be configured.

and does a 1.01 KB file take up 2KB?

Assuming a 1KB block size, yes that correct.

10
  • This is of course nonsense :-( The MMU is not involved here.
    – schily
    Nov 12 '19 at 15:38
  • 1
    @schily see the kernel docs for ext4. (Alpha could have 8K blocks thanks to its 8K pages.) Nov 12 '19 at 15:40
  • @StephenKitt SunOS 4.0 did handle file systems with a block size that did not match the MMU page size. It could just be slower. If you have that in mind, you should mention mmap() in your comment
    – schily
    Nov 12 '19 at 15:52
  • @StephenKitt Yes I understand that, but I don't see anything stating this as the reason. It draws attention to the memory page size matches the EXT3 block size and goes on to state this is good because it would be a problem if the block size exceeded the memory page size. But it falls short of giving memory page size as the reason for the default 4KB block size. Nov 12 '19 at 16:10
  • 1
    @Philip I see your point. I think 4K is the default because that’s the largest supported block size (excluding Alpha), and that given the size of disks nowadays, most users wouldn’t want smaller block sizes as a default. The cap on the block size is the memory page size. So one could say it’s a combination of factors: increasing disk sizes meeting the memory page size. My archives suggest that 4K was the default size before 2008 because of bugs with blocks < 4K, but I can’t find references to the bugs in question. Nov 12 '19 at 16:48
0

The average space used for a single file in the file system in your example is 4200 bytes.

The average size of a file in your example is 800 bytes, since the file overhead in a tar archive is 512 bytes.

Your claim could be true in case that many files are smaller than 800 bytes and some files are a bit larger than 4096 bytes and the filesystem uses an internal block size of 4096 bytes, but does not implement fragments that are smaller than 4096 bytes.

If all files would have been smaller than 4096 bytes, du would report 4G, so some of the files seem to need 2 blocks à 4096 bytes.

Not the answer you're looking for? Browse other questions tagged or ask your own question.