10

According to Wikipedia, ZFS has the following limits:

  • Max. volume size: 256 trillion yobibytes (2128 bytes)
  • Max. file size: 16 exbibytes (264 bytes)
  • Max. number of files:
    • Per directory: 248
    • Per file system: unlimited
  • Max. filename length: 255 ASCII characters (fewer for multibyte character encodings such as Unicode)

Why does it have these limits? What internally limits these things? Why couldn't ZFS have a theoretically unlimited volume size, or filename length, and so on?

0
31

What internally limits these things?

Long answer

ZFS's limits are based on fixed-size integers because that's the fastest way to do arithmetic in a computer.

The alternative is called arbitrary-precision arithmetic, but it's inherently slow. This is why arbitrary-precision arithmetic is an add-on library in most programming languages, not the default way of doing arithmetic. There are exceptions, but these are usually mathematics-oriented DSLs like bc or Wolfram Language.

If you want fast arithmetic, you use fixed-size words, period.

The speed hit from arbitrary precision arithmetic is bad enough inside a computer's RAM, but when a filesystem doesn't know how many reads it needs to make in order to load all of the numbers it needs into RAM, that would be very costly. A filesystem based on arbitrary sized integers would have to piece each number together from multiple blocks, requiring a lot of extra I/O from multiple disk hits relative to a filesystem that knows up front how big its metadata blocks are.

Now let's discuss the practical import of each of those limits:

Max. volume size

2128 bytes is effectively infinite already. We can write that number instead as roughly 1038 bytes, which means in order to hit that limit, you'd have to have a single Earth-sized ZFS pool where every one of its 1050 atoms is used to store data, and each byte is stored by an element no larger than 1012 atoms.

1012 atoms sounds like a lot, but it's only about 47 picograms of silicon.

The density of data in grams is 2.5×10-13 g/byte for microSD storage, as of this writing: the largest available SD card is 1 TB, and it weighs about 0.25g.¹ A microSD card isn't made of pure silicon, but you can't ignore the packaging, because we'll need some of that in our Earth-computer, too; we'll assume that the low density of the plastic and the higher density of the metal pins average out to about the same density as silicon. We also need some slop here to account for inter-chip interconnects, etc.

A pico-anything is 10-12, so our 47 pg and 2.5×10-13 g/B numbers above are about an order of magnitude apart. That means that to a first approximation, to construct a single maximally-sized ZFS pool out of the current largest-available microSD cards, you might have to use an entire Earth-sized planet's worth of atoms, and then only if you start off with something close to the right mix of silicon, carbon, gold, etc. such that you don't end up with so much slag that you blow the estimate.

If you think it's unfair that I'm using flash storage here instead of something denser like tape or disk, consider the data rates involved, as well as the fact that we haven't even tried to consider redundancy or device replacement. We have to assume that this Earth-sized ZFS pool will be composed of vdevs that never need replacing, and that they can transfer data fast enough that you can fill the pool in a reasonable time. Only solid-state storage makes sense here.

The approximation above is quite rough, and storage densities continue to climb, but keep things in perspective: in the future, to pull off this stunt of constructing maximally-sized ZFS pools, we'll still need to use the total crust-to-core resources of small planets.

Max. file size

So we've got a filesystem the size of a planet now. What can we say about the size of the files stored within it?

Let's give every person on the planet their own equally-sized slice of that pool:

1038 ÷ 1010 ≈ 1028 ÷ 1019 ≈ 109

That's the size of the pool divided by the population of Earth² divided by the maximum file size, in round numbers.

In other words, every person can store about a billion maximally-sized files in their tiny personal slice of our Earth-sized ZFS storage array.

(If it's bothering you that our storage array is still the size of a planet here in this example, remember that it had to be that big in order to hit the first limit above, so it is fair to continue to use it for this example here.)

That per-file maximum file size is 16 EiB under ZFS, which is 16× larger than the maximum volume size of ext4, which is considered ridiculously large today in its own right.

Imagine someone using their slice of Planet ZFS (formerly known as Earth) to store backups of maximally-sized ext4 disk images. Further, this demented customer (there's always one) has decided to tar them up, 16 per file, just to hit the ZFS maximum file size limit. Having done so, that customer will still have room to do that again about a billion more times.

If you're going to worry about this limit, that's the sort of problem you have to imagine needing to solve. And that's without even getting into the required data bandwidth needed to transfer that file to the online backup service once.

Let's also be clear about how improbable that Earth-computer is. First you'd have to figure out how to construct it without allowing it to collapse in on itself under the force of gravity and become molten at the center. Then you'd have to figure out how to manufacture it using every single atom on Earth without any leftover slag.

Now, since you've turned the surface of the Earth-computer into a hellscape, all the people trying to make use of that computer would have to live somewhere else, a place where you'd frequently hear people cursing the speed-of-light delays that add latency to every transaction between between the Earth-computer and wherever they live now. If you think your ~10ms Internet ping time is a problem today, imagine putting 2.6 light-seconds between your keyboard and the computer if we move the population of Earth to the moon so we can make this Earth-computer.

ZFS's volume and file size limitations are science fiction big.

Max. number of files per directory

248 is roughly 1014 files per directory, which is only going to be a problem for applications that try to treat ZFS as a flat filesystem.

Imagine an Internet researcher who is storing files about each IP address on the Internet. Let's say there are exactly 232 IPs being tracked after first subtracting the slack spaces in the old IPv4 space and then adding in the hosts now using IPv6 addresses to make the arithmetic come out nice. What problem is this researcher trying to tackle which requires him to construct a filing system that can store more than 216 — 65536! — files per IP?

Let's say this researcher is storing files per TCP port as well, so that with just one file per IP:port combination, we've eaten up our 216 multiplier.

The fix is simple: store the per-IP files in a subdirectory named after the IP, and store the per-port files in a subdirectory of the directory holding the per-IP files. Now our researcher can store 1014 files per IP:port combination, sufficient for a long-term global Internet monitoring system.

ZFS's directory size limit isn't what I'd call "science fiction big," as we know of real applications today that can hit this limit, but the power of hierarchy means you can just add another directory layer if you run up against the limit.

This limit is probably set as low as this purely to avoid making the data structures needed to find files in a given directory too big to fit into RAM. It encourages you to organize your data hierarchically to avoid this problem in the first place.

Max. filename length

While this one limit does seem stringent, it actually makes sense.

This limit doesn't originate with ZFS. I believe it dates back to FFS in 4.2BSD. I can't find the quote, but when this limit was young, someone pointed out that this is enough space for "a short letter to grandma."

So, that begs the question: why do you need to name your files more descriptively than that? Any true need greater than that probably calls for hierarchy, at which point you multiply the limit by the number of levels in the hierarchy, plus one. That is, if the file is buried 3 levels deep in the hierarchy, the limit on the name of the full path is 4 × 255 = 1020 characters.

Ultimately, this limit is a human limit, not a technological limit. File names are for the human's use, and humans really don't need more than 255 characters to usefully describe the content of a file. A higher limit simply wouldn't be helpful. The limitation is old (1983) because humans haven't acquired the ability to cope with longer file names since then.

If you're asking where the odd-looking "255" value comes from, it's some limitation based on the size of an 8-bit byte. 28 is 256, and the N-1 value used here probably means they're using a null terminator to mark the end of the file name string in a 256-byte field in the per-file metadata.

Short answer

Practically speaking, what limits?


Footnotes:

  1. I measured this using a scale specified with an accuracy of 0.01g.

  2. 7.55 billion, as of this writing. Above, we're rounding this off to 1010, which we should hit by mid-century.

5
  • 4
    Enjoyable read, thanks! The minimum number for PATH_MAX on a POSIX system is 256. This can be made up of components of at most NAME_MAX characters each (this value is at least 14). – Kusalananda Jan 12 '17 at 21:04
  • 2
    Very good answer. To add to the filename part: Long filenames actually decrease usability for humans, especially if mixed with short names (more screen size needed to display them, layout will be affected, shell history will be harder to read etc.), and they are still inferior to a flexible and searchable tagging system (which ZFS lacks, unfortunately). – user121391 Jan 13 '17 at 12:23
  • That's amazing, but why did they cripple the filename to 255 characters? There are very practical use cases for that, e.g. long course or book or paper titles along with by the list of author names. And there's software that breaks when it can't write the complete filename, e.g. youtube-dl when downloading the video of such a course. – Dan Dascalescu Nov 4 '19 at 6:05
  • @DanDascalescu I justified that in the answer and gave remedies. – Warren Young Dec 4 '19 at 5:47
  • @WarrenYoung: no need to justify, since you didn't impose the limit. However, I don't feel that as it is, the "Max. filename length" section addresses my objection (with the "course/book/paper" title example). I want my book/course/video filename t obe self-sufficient, not artificially split into a directory (e.g. the author) plus a filename. See the zero, one, infinity rule and run a simple search for "filename too long" -windows - it reveals tens of millions of results. – Dan Dascalescu Dec 5 '19 at 0:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.