61

I have a directory tree that contains many small files, and a small number of larger files. The average size of a file is about 1 kilobyte. There are 210158 files and directories in the tree (this number was obtained by running find | wc -l).

A small percentage of files gets added/deleted/rewritten several times per week. This applies to the small files, as well as to the (small number of) larger files.

The filesystems that I tried (ext4, btrfs) have some problems with positioning of files on disk. Over a longer span of time, the physical positions of files on the disk (rotating media, not solid state disk) are becoming more randomly distributed. The negative consequence of this random distribution is that the filesystem is getting slower (such as: 4 times slower than a fresh filesystem).

Is there a Linux filesystem (or a method of filesystem maintenance) that does not suffer from this performance degradation and is able to maintain a stable performance profile on a rotating media? The filesystem may run on Fuse, but it needs to be reliable.

Improve this question
10
  • If you know which files are going to be big/not changing very often, and which are going to be small/frequently changing, you might want to create two filesystems with different options on them, more suited to each scenario. If you need them to be accessible as they were a part of the same structure, you can do some tricks with mount, symlinks.
    – Marcin
    Jan 10, 2012 at 13:22
  • I am quiet surprised to know that btrfs(with copy-on-write feature) has been sluggish to you over a period of time. I am curious to have the results shared from you, possibly helping each other in new direction of performance tuning with it.
    – Nikhil Mulley
    Jan 10, 2012 at 14:00
  • there is a new animal online zfs on Linux, available in native mode and fuse implementations, incase you wanted to have a look.
    – Nikhil Mulley
    Jan 10, 2012 at 14:03
  • I tried zfs on linux once, was quite unstable. Managed to completely lock up the filesystem quite often. Box would work, but any access to the FS would hang.
    – phemmer
    Jan 10, 2012 at 14:46
  • Similar post serverfault.com/questions/6711/…
    – Nikhil Mulley
    Jan 10, 2012 at 15:22

4 Answers 4

Reset to default
63

Performance

I wrote a small Benchmark (source), to find out, what file system performs best with hundred thousands of small files:

  • create 300000 files (512B to 1536B) with data from /dev/urandom
  • rewrite 30000 random files and change the size
  • read 30000 sequential files
  • read 30000 random files

  • delete all files

  • sync and drop cache after every step

Results (average time in seconds, lower = better):

Using Linux Kernel version 3.1.7
Btrfs:
    create:    53 s
    rewrite:    6 s
    read sq:    4 s
    read rn:  312 s
    delete:   373 s

ext4:
    create:    46 s
    rewrite:   18 s
    read sq:   29 s
    read rn:  272 s
    delete:    12 s

ReiserFS:
    create:    62 s
    rewrite:  321 s
    read sq:    6 s
    read rn:  246 s
    delete:    41 s

XFS:
    create:    68 s
    rewrite:  430 s
    read sq:   37 s
    read rn:  367 s
    delete:    36 s

Result:
While Ext4 had good overall performance, ReiserFS was extreme fast at reading sequential files. It turned out that XFS is slow with many small files - you should not use it for this use case.

Fragmentation issue

The only way to prevent file systems from distributing files over the drive, is to keep the partition only as big as you really need it, but pay attention not to make the partition too small, to prevent intrafile-fragmenting. Using LVM can be very helpful.

Further reading

The Arch Wiki has some great articles dealing with file system performance:

https://wiki.archlinux.org/index.php/Beginner%27s_Guide#Filesystem_types

https://wiki.archlinux.org/index.php/Maximizing_Performance#Storage_devices

Improve this answer
27
  • 4
    You should specify what version of the kernel youre basing that comparison off of. XFS got some very significant speed imporovments in one of the recent kernels (think it was 2.6.31, but dont quote me on that).
    – phemmer
    Jan 11, 2012 at 1:47
  • 1
    btrfs internally does your lvm trick. It allocates smaller chunks of the disk and places files in those chunks, then only allocates another chunk of the disk when the existing chunks fill up.
    – psusi
    Jan 11, 2012 at 19:38
  • 1
    That's true of any filesystem. That is why applications use things like fsync().
    – psusi
    Jan 12, 2012 at 18:49
  • 2
    @taffer, it is. The transactions have the same effect as the journal does in other filesystems: they protect the fs metadata. In theory they can be used by applications in the way you describe, but there is currently no api to allow applications to open and close transactions.
    – psusi
    Jan 12, 2012 at 20:31
  • 1
    @taffer Your "recent benchmark" is from April 2015, over three years old and uses XFS with only default options. This pre-dates xfsprogs 3.2.3 which makes XFS v5 the default and all the benefits it brings. It also wasn't formatted with -m finobt=1 which is a game-changer for XFS performance with small files and heavy metadata updates. No, there are no silver bullets, but basing your opinion on old benchmarks with is not wise, especially when major performance-changing features were ignored, unavailable, or disabled.
    – Jody Bruchon
    Aug 23, 2018 at 21:14
9

The ext4 performance drops off after 1-2 million files in a directory. See this page http://genomewiki.ucsc.edu/index.php/File_system_performance created by Hiram Clawson at UCSC

Improve this answer
8

I am using ReiserFS for this task, it is especially made for handling a lot of small files. There is an easy to read text about it at the funtoo wiki.

ReiserFS also has a host of features aimed specifically at improving small file performance. Unlike ext2, ReiserFS doesn't allocate storage space in fixed one k or four k blocks. Instead, it can allocate the exact size it needs.

Improve this answer
1
  • 1
    There are stability issues as well with ReiserFS - so RH and SuSE have dropped that FS. From the principle (BTree-based-FS) BTRFS should be comparable.
    – Nils
    Jan 10, 2012 at 20:59
-1

XFS is noted for performing very well in situations like this. This is part of why we use it at my work for our mail stores (that can contain hundreds of thousands of files in 1 directory). It has better fault tolerance than ReiserFS, is in much wider use, and is generally a very mature filesystem.

Additionally, XFS supports online defragmentation. Though it does use a delayed allocation technique which results in less fragmentation (vs other filesystems) to begin with.

Improve this answer
9
  • 27
    XFS is noted for performing very well in situations like this. [citation needed]
    – taffer
    Jan 14, 2012 at 16:13
  • 11
    Ehm, xfs is especially known for the opposite: working really well with big files, but not that well on small ones! Look at this exhaustive benchmark for example (or jump right to the conclusion on page 10 ^^): ilsistemista.net/index.php/linux-a-unix/…
    – Levite
    Jul 2, 2014 at 12:51
  • 1
    @Levit I think you're misreading that report. The report very clearly shows that XFS performs very well for random IO. But that aside, the report does not address the type of scenario in this question, lots of files. Random IO is one thing, large numbers of files is where ext* falls on its face.
    – phemmer
    Jul 2, 2014 at 13:26
  • 2
    The only place XFS is really better there are the random read/write operations (still seems strange that a truly random read pattern on a mechanical disk is able to get 10MB/s - seems to me like some optimization that does not fly in real world (imho)), whereas on page 7 it shows just what I said earlier, XFS is really good in handling big files! Look at pages 3&5, esp on 3 you see it handling small files clearly not as well as ext! I really don't have anything against XFS though, but from what you find just about everywhere, it is not the best optiom for many small files, is all I am saying!
    – Levite
    Jul 3, 2014 at 6:22
  • 5
    XFS also can be extremely slow when it comes to big files, if these files are extended randomly/slowly with small chunks over a long time. (The typical syslogd pattern.) For example at my side in an XFS over MD setup I just observed, that removing a 1.5 GB file took 4.75 minutes(!) while the disk drive was capped at a 100 transactions/s limit at a write rate of more than 2 MB/s. This also impacts performance of other parallel going IO operations on the same drive badly, as the drive already is maxed out. Never saw anything like that in other FS (or being tested in benchmarks).
    – Tino
    Jun 13, 2015 at 12:40

You must log in to answer this question.