Could dpkg, the Debian package manager, gain a noticeable performance improvement by using one of the AIO fsync() operations, instead of sync_file_range() + fsync()?
The [proposed] fsync2() API is essentially identical to the existing AIO_FSYNC/AIO_FDSYNC API, except it's synchronous and that is what applications want to avoid.
The only argument I've been presented with against [using] AIO_FSYNC is that "the implementation is just a workqueue", which is largely non-sensical because it is filesystem implementation independent but allows automatic kernel side parallelisation of all the fsync operations issued. This allows the filesystem(s) to then automatically optimise away unnecessary journal writes when completing concurrent fsync operations - XFS, ext4, etc already do this when user applications run fsync() concurrently from lots of processes/threads.....
This simple implementation allows a simple "untar with aio fsync" workload (i.e."write many 4kB files and aio_fsync() in batches as we go, retiring completed fsync()s before we dispatch a new batch") workload on XFS to go from about 2000 files/s (synchronous write IO latency bound) to over 40,000 files/s (write iops bound on the back end storage).
-- Dave Chinner
The example workload has similarities with apt-get install or dpkg -i (partly depending on the size of files in the installed packages :-). dpkg must effectively fsync() all unpacked files, before it renames them into place.
dpkg has been optimized using advice from Ted T'so. The optimization is to add calls to sync_file_range() at certain points. This system call does not provide the same guarantees as fsync(). Please read the documentation for sync_file_range() and notice the prominent warning :-).
None of these operations writes out the file's metadata. Therefore, unless the application is strictly performing overwrites of already-instantiated disk blocks, there are no guarantees that the data will be available after a crash.
dpkg triggers data writeback immediately after writing each file, using SYNC_FILE_RANGE_WRITE. It writes all the files for the package first. Then there is a second pass through the files, which waits for the data writeback using SYNC_FILE_RANGE_WAIT_BEFORE, calls fsync(), and finally renames the file into place.
See commits:
- Disable usage of synchronous sync(2) by default
- Add new --force-unsafe-io to disable safe I/O operations on unpack
- On Linux initiate writeback of unpacked files ASAP
- On Linux finish writeback before fsync
My hypothesis is that parallelising the fsync() operations instead could improve performance, by allowing more efficient batching of the metadata writes, particularly batching the associated barriers/disk cache flushes which are required to ensure on-disk metadata is consistent at all times.
EDIT: It seems my hypothesis was too simple, at least when using the ext4 filesystem:
The second series of sync_file_range() calls, with the operation
SYNC_FILE_RANGE_WAIT_BEFORE, will block until the previously initiated writeback has completed. This basically ensures that the delayed allocation has been resolved; that is, the data blocks have been allocated and written, and the inode updated (in memory), but not necessarily pushed out to disk.The [fsync()] call will actually force the inode to disk. In the case of the ext4 file system, the first [fsync()] will actually push all of the inodes to disk, and all of the subsequent [fsync()] calls are in fact no-ops (assuming that files 'a', 'b', and 'c' are all on the same file system). But what it means is that it minimizes the number of (heavyweight) jbd2 commits to a minimum.
It uses a linux-specific system call --- sync_file_range() --- but the result should be faster performance across the board for all file systems. So I don't consider this an ext4-specific hack, although it probably does makes things faster for ext4 more than any other file system.
-- Ted T'so
It might be that some other filesystem would benefit using AIO fsync() operations instead.
bcachefs (under development) claims to isolate IO between different files much better than ext4. So that might be particularly interesting to test.
It sounds as if ext4 might not be so well-optimized for a pure AIO fsync() pattern (I guess other filesystems could have the same constraint as well). If so, I suppose it would be possible to do all the same sync_file_range() calls first, then start off all the AIO fsync() operations as a second round, and finish up by renaming all the files into place as the fsync() operations complete.
OLD:
The first step in such an investigation should be measurement :-).
It is possible to disable the fsync() part, using echo "force-unsafe-io" > /etc/dpkg/dpkg.cfg.d/force-unsafe-io.
So far, I tried running apt-get install under strace -f -wc, in a Debian 9 container. E.g. installing the aptitude package using "unsafe io", there are only 495 synchronous fsync() calls. Whereas installing aptitude normally, there are 1011 fsync() calls. "unsafe io" also disabled the SYNC_FILE_RANGE_WAIT_BEFORE call, reducing the number of sync_file_range() calls from 1036 to 518.
However, it was much less clear whether this reduced the average time taken. If it did, it does not seem to be by more than the random variation between runs. So far, I tested this on ext4 and XFS, on a mechanical HDD.
apt-get says the total size of the 518 unpacked files was 21.7 MB (see output below).
Regarding the 495 fsync() calls, which remained present even when requesting "unsafe io":
On ext4, the strace output showed the time spent on the remaining fsync() calls as about 11 seconds. On XFS, the corresponding figure was about 7 seconds. In all cases, this was the majority of the time taken to install aptitude.
So even if "unsafe io" is giving a small improvement for installing aptitude, it seems like you would need /var to be mounted on a significantly faster (lower latency) device than the rest of the system, before the difference would be really noticeable. But I am not interested in optimizing that niche case.
Running under strace -f -y -e trace=fsync,rename showed that for the remaining fsync() calls, 2 of them were on /etc/ld.so.cache~, and 493 of them were to files inside /var/lib/dpkg/ i.e. the package database.
318 of the fsync() calls are under /var/lib/dpkg/updates/. These are increments to the dpkg database /var/lib/dpkg/status. The increments are rolled up into the main database ("checkpointed") at the end of the dpkg run.
The following NEW packages will be installed:
aptitude aptitude-common libboost-filesystem1.62.0 libboost-iostreams1.62.0 libboost-system1.62.0 libcgi-fast-perl libcgi-pm-perl
libclass-accessor-perl libcwidget3v5 libencode-locale-perl libfcgi-perl libhtml-parser-perl libhtml-tagset-perl libhttp-date-perl
libhttp-message-perl libio-html-perl libio-string-perl liblwp-mediatypes-perl libparse-debianchangelog-perl libsigc++-2.0-0v5 libsqlite3-0
libsub-name-perl libtimedate-perl liburi-perl libxapian30
0 upgraded, 25 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/6000 kB of archives.
After this operation, 21.7 MB of additional disk space will be used.
