Heavy write activity on SSD nukes system performance

Question

I've noticed that when I do heavy write applications, the whole system slows down. To test this further I ran this to do a (relatively) low-CPU, high disk activity:

john -incremental > file_on_SSD

This pumps out tens of thousands of strings per second to a file on my system disk.

When it's doing this, the mouse lags, TTYs become unresponsive, applications "fade" and generally the whole computer becomes unusable. When I can eventually Control+C john, the system comes back to full strength after a few seconds.

This is an extreme example but I have similar issues with slightly less write-intensive activities like copying big files from fast sources or transcoding.

My main OS disk is a quite fast SSD (OCZ Agility 60GB) with EXT4. If I write john's output to a mechanical disk with EXT4, I don't experience the same slow-downs though the rate is a lot slower (SSD does ~42,000 words per second, mechanical does 8,000 w/s). The throughput may be relevant. The mechanical disk also has nothing to do with the system. It's just data.

And I'm using kernel 2.6.35-2 but I've noticed this issue since I got this SSD when I was probably using .31 or something from around that time.

So what's causing the slowdown? EXT4 issue? Kernel issue? SSD issue? All of the above? Something else?

If you think I need to run an additional test, just drop a comment telling me what to do and I'll append the result to the question.

Answer 1

This has been a known issue for awhile. Using an SSD-tuned FS like Btrfs might help, but it might not.

Ultimately, it is a bug in the IO scheduler/memory management systems. Recently, there have been some patches that aim to address this issue. See Fixed: The Linux Desktop Responsiveness Problem?

These patches may eventually make their way into the mainline kernel, but for now, you will probably have to compile your own kernel if you want to fix this issue.

Answer 2

There are a few things you can check on to try to improve SSD performance under Linux.

1. Set the mount point to 'noatime'. Extra activity updating access times are generally wasted on most use cases. Especially in the case of continually pumping single lines into a file, you are forcing multiple updates to the filesystem for every access.

2. Check the elevator. The default elevator for most distros are set up for random access spinning platters. SSD's don't need the extra logic, so setting the elevator to noop can improve performance by letting the hardware manage the writes.

3. Write-through v write-back caching. This is a bit more esoteric, but you can check the caching method used with hdparm for the device. Write-back caching can have a positive impact on SSD performance compared to write-through.

Answer 3

Your file caching is probably incorrectly tuned for your workload. Unfortunately, Linux kernel is stupid enough to not handle this automatically and the defaults are pretty bad if you have lots of RAM and slow enough block devices. See https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/ for details.

I'd suggest trying modifying /etc/sysctl.conf either

vm.dirty_background_ratio = 3
vm.dirty_ratio = 6

to heavily reduce RAM pressure caused by write caching to allow kernel to better handle other tasks. This will trade improved latency for lower throughput.

Another possibility is to increase caching but if your process keeps constantly writing new data all the time you will hit very bad latency if the cache ever gets full. If you want to try it, you can do something like

vm.dirty_background_ratio = 5
vm.dirty_ratio = 80

Note that *_ratio settings refer to percentage of available RAM. If you want better control, use *_bytes settings. I personally use following config for my workstation:

vm.dirty_background_bytes = 50000000
vm.dirty_bytes = 200000000

That limits background writing cache to 50 MB and forces syncronous writing if 200 MB is in the cache.