1

Could dpkg, the Debian package manager, gain a noticeable performance improvement by using one of the AIO fsync() operations, instead of sync_file_range() + fsync()?

The [proposed] fsync2() API is essentially identical to the existing AIO_FSYNC/AIO_FDSYNC API, except it's synchronous and that is what applications want to avoid.

The only argument I've been presented with against [using] AIO_FSYNC is that "the implementation is just a workqueue", which is largely non-sensical because it is filesystem implementation independent but allows automatic kernel side parallelisation of all the fsync operations issued. This allows the filesystem(s) to then automatically optimise away unnecessary journal writes when completing concurrent fsync operations - XFS, ext4, etc already do this when user applications run fsync() concurrently from lots of processes/threads.....

This simple implementation allows a simple "untar with aio fsync" workload (i.e."write many 4kB files and aio_fsync() in batches as we go, retiring completed fsync()s before we dispatch a new batch") workload on XFS to go from about 2000 files/s (synchronous write IO latency bound) to over 40,000 files/s (write iops bound on the back end storage).

-- Dave Chinner

The example workload has similarities with apt-get install or dpkg -i (partly depending on the size of files in the installed packages :-). dpkg must effectively fsync() all unpacked files, before it renames them into place.

dpkg has been optimized using advice from Ted T'so. The optimization is to add calls to sync_file_range() at certain points. This system call does not provide the same guarantees as fsync(). Please read the documentation for sync_file_range() and notice the prominent warning :-).

None of these operations writes out the file's metadata. Therefore, unless the application is strictly performing overwrites of already-instantiated disk blocks, there are no guarantees that the data will be available after a crash.

dpkg triggers data writeback immediately after writing each file, using SYNC_FILE_RANGE_WRITE. It writes all the files for the package first. Then there is a second pass through the files, which waits for the data writeback using SYNC_FILE_RANGE_WAIT_BEFORE, calls fsync(), and finally renames the file into place.

See commits:

My hypothesis is that parallelising the fsync() operations instead could improve performance, by allowing more efficient batching of the metadata writes, particularly batching the associated barriers/disk cache flushes which are required to ensure on-disk metadata is consistent at all times.

EDIT: It seems my hypothesis was too simple, at least when using the ext4 filesystem:

The second series of sync_file_range() calls, with the operation SYNC_FILE_RANGE_WAIT_BEFORE, will block until the previously initiated writeback has completed. This basically ensures that the delayed allocation has been resolved; that is, the data blocks have been allocated and written, and the inode updated (in memory), but not necessarily pushed out to disk.

The [fsync()] call will actually force the inode to disk. In the case of the ext4 file system, the first [fsync()] will actually push all of the inodes to disk, and all of the subsequent [fsync()] calls are in fact no-ops (assuming that files 'a', 'b', and 'c' are all on the same file system). But what it means is that it minimizes the number of (heavyweight) jbd2 commits to a minimum.

It uses a linux-specific system call --- sync_file_range() --- but the result should be faster performance across the board for all file systems. So I don't consider this an ext4-specific hack, although it probably does makes things faster for ext4 more than any other file system.

-- Ted T'so

It might be that some other filesystem would benefit using AIO fsync() operations instead.

bcachefs (under development) claims to isolate IO between different files much better than ext4. So that might be particularly interesting to test.

It sounds as if ext4 might not be so well-optimized for a pure AIO fsync() pattern (I guess other filesystems could have the same constraint as well). If so, I suppose it would be possible to do all the same sync_file_range() calls first, then start off all the AIO fsync() operations as a second round, and finish up by renaming all the files into place as the fsync() operations complete.


OLD:

The first step in such an investigation should be measurement :-).

It is possible to disable the fsync() part, using echo "force-unsafe-io" > /etc/dpkg/dpkg.cfg.d/force-unsafe-io.

So far, I tried running apt-get install under strace -f -wc, in a Debian 9 container. E.g. installing the aptitude package using "unsafe io", there are only 495 synchronous fsync() calls. Whereas installing aptitude normally, there are 1011 fsync() calls. "unsafe io" also disabled the SYNC_FILE_RANGE_WAIT_BEFORE call, reducing the number of sync_file_range() calls from 1036 to 518.

However, it was much less clear whether this reduced the average time taken. If it did, it does not seem to be by more than the random variation between runs. So far, I tested this on ext4 and XFS, on a mechanical HDD.


apt-get says the total size of the 518 unpacked files was 21.7 MB (see output below).

Regarding the 495 fsync() calls, which remained present even when requesting "unsafe io":

On ext4, the strace output showed the time spent on the remaining fsync() calls as about 11 seconds. On XFS, the corresponding figure was about 7 seconds. In all cases, this was the majority of the time taken to install aptitude.

So even if "unsafe io" is giving a small improvement for installing aptitude, it seems like you would need /var to be mounted on a significantly faster (lower latency) device than the rest of the system, before the difference would be really noticeable. But I am not interested in optimizing that niche case.

Running under strace -f -y -e trace=fsync,rename showed that for the remaining fsync() calls, 2 of them were on /etc/ld.so.cache~, and 493 of them were to files inside /var/lib/dpkg/ i.e. the package database.

318 of the fsync() calls are under /var/lib/dpkg/updates/. These are increments to the dpkg database /var/lib/dpkg/status. The increments are rolled up into the main database ("checkpointed") at the end of the dpkg run.


The following NEW packages will be installed:
  aptitude aptitude-common libboost-filesystem1.62.0 libboost-iostreams1.62.0 libboost-system1.62.0 libcgi-fast-perl libcgi-pm-perl
  libclass-accessor-perl libcwidget3v5 libencode-locale-perl libfcgi-perl libhtml-parser-perl libhtml-tagset-perl libhttp-date-perl
  libhttp-message-perl libio-html-perl libio-string-perl liblwp-mediatypes-perl libparse-debianchangelog-perl libsigc++-2.0-0v5 libsqlite3-0
  libsub-name-perl libtimedate-perl liburi-perl libxapian30
0 upgraded, 25 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/6000 kB of archives.
After this operation, 21.7 MB of additional disk space will be used.
1

The question suggests this won't help on ext4 or XFS.

I also tested with installing one much larger package (linux-image-4.9.0-9-amd64). It still seemed to take the same time, regardless of --force-unsafe-io.

ext2

On ext2, --force-unsafe-io reduced the time to install linux-image from 50 seconds to 13 seconds.

The kernel I ran the tests on was 5.0.17-200.fc29.x86_64, which uses CONFIG_EXT4_USE_FOR_EXT2.

I tested ext2 using the userspace aio_fsync() implementation. However, the best improvement did not depend on using AIO fsync().

My improvement was actually due to a side-effect. I had changed dpkg to do all of the fsync() operations first, and then all of the rename() operations. Whereas the unpatched dpkg called rename() after each fsync(). I used AIO queue depths of up to 256. AIO fsync() with a queue depth of 1 was significantly slower than synchronous fsync() - it appears there was some overhead. The best improvement also required doing all the original SYNC_FILE_RANGE_WRITE operations first. The improved version installed linux-image in about 18 seconds.

This order of operations is actually what Ted T'so originally suggested :-D. What happens is that on CONFIG_EXT4_USE_FOR_EXT2, fsync() also helpfully syncs the parent directory. You want to do all the file name manipulation first, so you can avoid multiple on-disk updates for each directory. I think this does not happen for the old CONFIG_EXT2 implementation, or for a normal ext4 filesystem.

ext4: make fsync to sync parent dir in no-journal for real this time

[...] This also includes ext2 default mode obviously. [...]

https://elixir.bootlin.com/linux/v5.0.17/source/fs/ext4/fsync.c#L38

 * If we're not journaling and this is a just-created file, we have to
 * sync our parent directory (if it was freshly created) since
 * otherwise it will only be written by writeback, leaving a huge
 * window during which a crash may lose the file.  This may apply for
 * the parent directory's parent as well, and so on recursively, if
 * they are also freshly created.

As before, replacing the fsync() stage with sync() appears to give disturbingly good performance, matching --force-unsafe-io :-). sync() or syncfs() seem to be very nice if you can get away with using them.

btrfs

When I started testing aio_fsync() on btrfs, I discovered that fsync() operations can cause rename() of the file to block, due to a recent data integrity fix. I decided I am not interested in btrfs.

Why does rename() take longer when fsync() is called first?

  • Nice investigation; are you planning to pursue this with the dpkg maintainer? – Stephen Kitt Jun 4 at 7:57
  • @StephenKitt notice that per the question, this was not a massive benefit unless you are unpacking a significant amount of "larger" packages. For "smaller" packages, the majority of the time was spent syncing the dpkg database. – sourcejedi Jun 4 at 8:42
  • Yes, indeed, I had noticed that; the benefit is somewhat limited in scope, but if the code doesn’t end up being too complex and the “base case” doesn’t slow down... (Obviously it’s still work, so not free in any case.) – Stephen Kitt Jun 4 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.