3

My main desktop has 64GB of RAM. This seems to cause the dirty write cache to often exceed 10GB during disk-intensive operations such as file copies or network transfers. (I gathered this by observing the Dirty value in /proc/meminfo before a transfer - either 0 or very close to it - and during a transfer - over 10GB) I boot from an SSD, but work with large data transfers on external hard drives. One example external drive can probably write at around 100MB/sec average, but I'm connected to my ZFS-array-based NAS via 10Gbit networking, so >400MB/sec bandwidth via the LAN is common.

The problem with this is that if I am multitasking, and if another task executes a sync operation (one example is I think apt or dpkg and similar execute a sync right before completing), it can hang the entire process for several minutes. Additionally, the writing process is still writing, so the sync can take even longer than just the time to empty the dirty cache. This also is evident on the system in other areas - random lags and hangs occur in applications during the transfer.

10GB is quite an insanely large write cache even for a hard drive. It might not be so bad with my NVMe boot SSD that can achieve >1GB/sec writes, but on a relatively slow hard drive it can cause the sync time to exceed 2 minutes assuming no new writes are occurring. If a very large ongoing transfer is occurring I've seen the sync command stall for >10 minutes!

Is there any way to tell the Linux kernel to limit how much data can be outstanding to a given device? (e.g. if I could tell the kernel to only allow, say, 512MB of data to be in cache prior to forcing data to start flushing to disk, that would help the situation immensely.) Failing that, I could maybe live with lowering the systemwide write cache, but also not sure how to do that.

3
  • Do you have any other media attached (e.g., some slow usb-based device)? A sync wouldn't be limited only to your hard drive.
    – Andy Dalton
    Mar 24, 2021 at 1:55
  • Actually that's the very issue. For example, if I need to run an apt command, it will do a sync. The sync will cause the buffer waiting on the external drive to get flushed, stalling the terminal (and lagging everything else) for >2 minutes. I want to reduce the buffer that the external drive uses, so that a full sync of all devices won't slow down the entire system (because there won't be as much data that needs to get flushed to the external disk during a sync of all devices).
    – fdmillion
    Mar 24, 2021 at 3:37
  • apt and dpkg call fsync() on specific files... they shouldn't be calling for a global sync.
    – psusi
    Mar 24, 2021 at 18:21

1 Answer 1

Reset to default
3

Per device write cache is yet to be implemented and the problem you're talking is well known and I voiced it over five years ago: Why were "USB-stick stall" problems reported in 2013? Why wasn't this problem solved by the existing "No-I/O dirty throttling" code?

I also have 64GiB of RAM and this is how I solved it for myself:

cat /etc/sysctl.d/dirty.conf
# Per Torvalds advice
vm.dirty_background_bytes = 33554432
vm.dirty_bytes = 134217728

It might make certain operations run a little bit slower but it will solve your issue.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .