6

I have a Debian 10 server with a 2TB drive where I want to store over 120 million small files. If I use ext4 I run out of inodes.

What file system should I use?

I've been reading about reiserfs and reiser4, but I'm not sure if either of them are still supported.

Is there a file system with a built Debian package that I can use?

The main use case is providing 256x256 raster map tiles to users with Apache. I'm guessing that speed isn't that important as the limiting factor will be the ping time. Using as much of the bytes as I can is mildly important, but I also have a lot of free disk space.

Edit: The files are typically 1kb to 3kb.

3
  • Use BTRFS or XFS, EXT4 is not suited for many small files.
    – paladin
    Sep 16, 2021 at 8:39
  • Could you add the approximate size per file to the question?
    – AdminBee
    Sep 16, 2021 at 10:32
  • While not an answer it just bits of data, should you not be using a data base? Setting up & organizing data would also make it easier to access.
    – oldfred
    Sep 16, 2021 at 14:48

2 Answers 2

6

You don't stipulate how small "small" is, but if the files are both sufficiently small and sufficiently compressible, ZFS pools with the embedded_data feature enabled can store "small" files in the block pointer itself, which eliminates the need to allocate a block (or more) of storage for the file, and also eliminates the I/O call to read or write the file, since the file data is written along with the file metadata into the block pointer itself.

Also, note that ZFS will never run out of inodes.

 embedded_data

           This feature improves the performance and compression ratio of
           highly-compressible blocks.  Blocks whose contents can compress
           to 112 bytes or smaller can take advantage of this feature.

           When this feature is enabled, the contents of highly-
           compressible blocks are stored in the block "pointer" itself (a
           misnomer in this case, as it contains the compressed data,
           rather than a pointer to its location on disk).  Thus the space
           of the block (one sector, typically 512 bytes or 4KB) is saved,
           and no additional i/o is needed to read and write the data
           block.
4

Assuming you want to optimise the usage of disk space (not only inode count and perhaps access times):

You may want a file system with block suballocation / tail merging so that small data are merged into one allocation unit ("cluster") on the physical disk.

Also, depending on your scenario it may be wise to control the cluster size of the file system, again to save space. The optimum could be determined by a test.

A stable candidate for Linux is btrfs.

Of course, if you are happy using ext4 (and perhaps not being space optimal) you could re-create your file system (copy the data elsewhere to back it up first!) with an option for very many inodes.

From mke2fs(8):

   -N number-of-inodes
          Overrides the default calculation of the number of inodes that
          should be reserved for the filesystem (which is based on the number
          of blocks and the  bytes-per-inode ratio).  This allows the user to
          specify the number of desired inodes directly.
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.