5

I have these seagate disks (ST5000LM000 - note they are SMR) that when I put them under heavy write workloads the i/o utilization goes to 100% and throughput basically goes to zero. The disks are plugged into a SAS controller that uses the mpt3sas driver (disks show up as scsi devices). I have tried changing to the noop scheduler, setting ncq to 1, and increasing the device timeout to 1 hour. I have even tried a completely different disk controller (which uses the megaraid driver) which did not change anything. Each drive has a single XFS partition.

The only thing that seems to help is decreasing the concurrency of my script writing out files so that disk i/o never gets behind so far that the snowball effect grinds things to a halt.

I thought echo 1 > /sys/block/sdl/device/queue_depth should prevent concurrent disk operations, but I commonly see around 150 operations in-flight from cat /sys/block/sdl/stat.

This is a big problem because if I don't kill the load script when this starts happening, eventually i/o operations time out causing the disk to disconnect, which sometimes causes a process to get stuck in the dreaded D state and I often end up with corrupt data.

Are there kernel settings I can change to prevent getting into this bad state? It seems like there should be something I can do since if I kill it early enough it can always get caught up before any i/o operations time out and disconnects a disk.

kern.log from when a disk actually disconnects

[401217.833235] sd 0:0:6:0: device_block, handle(0x0010)
[401218.583675] mpt3sas_cm0: log_info(0x31110e03): originator(PL), code(0x11), sub_code(0x0e03)
[401218.833518] sd 0:0:6:0: device_unblock and setting to running, handle(0x0010)
[401222.584105] sd 0:0:6:0: device_block, handle(0x0010)
[401230.581727] sd 0:0:6:0: device_unblock and setting to running, handle(0x0010)
[401230.586627] scsi_io_completion: 6 callbacks suppressed
[401230.586641] sd 0:0:6:0: [sdg] tag#0 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[401230.586656] sd 0:0:6:0: [sdg] tag#0 CDB: Read(16) 88 00 00 00 00 01 3b e5 74 18 00 00 02 00 00 00
[401230.586661] XFS (sdg): metadata I/O error: block 0x800007b8 ("xfs_trans_read_buf_map") error 5 numblks 32
[401230.586670] XFS (sdg): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
[401230.597537] blk_update_request: 6 callbacks suppressed
[401230.597540] blk_update_request: I/O error, dev sdg, sector 5299860504

disk bandwidth basically drops to zero disk bandwidth basically drops to zero average i/o request time skyrockets average i/o request time skyrockets disk i/o stays at 100% utilization disk i/o stays at 100% utilization in-flight i/o requests stays around 150 in-flight i/o requests stays around 150 (note that in the above images it recovers in the end because I cancel my load script somewhere in the middle of when write throughput significantly drops)

distro/kernel

$ lsb_release -d
Description:    Ubuntu 16.04.6 LTS
$ uname -r
4.15.0-62-generic

fdisk -l

Disk /dev/sdl: 4.6 TiB, 5000981078016 bytes, 9767541168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

xfs_info

meta-data=/dev/sdl               isize=512    agcount=5, agsize=268435455 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1 spinodes=0
data     =                       bsize=4096   blocks=1220942646, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
5
  • How did you increase the timeout? Just want to confirm, as increase the timeout is a darn good first idea on how to fix. And that RedHat link has a clear mistake: they lost a > somehow (should be echo value > /sys/block/device-name/device/timeout).
    – derobert
    Sep 12, 2019 at 19:27
  • @derobert yes, that is what I did, but I am not sure it actually works (sometimes a drive will disconnect well under an hour) maybe there is some other timeout I am missing? ``` $ cat /sys/block/sdl/device/timeout 3600 ```
    – gregjhogan
    Sep 12, 2019 at 19:30
  • Also, which kernel version are you running? I personally had issues with SATA SMR disks that were fixed with a kernel upgrade. That was a good while ago, though (and unfortunately it'd take a lot of work to figure out what version. Bet it was 3.something though.).
    – derobert
    Sep 12, 2019 at 19:35
  • @derobert 4.4.0-161-generic (added to question)
    – gregjhogan
    Sep 12, 2019 at 21:09
  • updated to 4.15.0-62-generic kernel and no change in behavior
    – gregjhogan
    Sep 13, 2019 at 19:23

2 Answers 2

3

I change the below kernel params and the SMR disks no longer disconnect under heavy write load. Sometimes write performance will slow to a craw under heavy I/O (like single digit MB/s write speeds) but disks at least don't disconnect any longer.

DEVICE=sdX # insert your device name here
echo 3600 > /sys/block/$DEVICE/device/timeout
echo 3600 > /sys/block/$DEVICE/device/eh_timeout
echo noop > /sys/block/$DEVICE/queue/scheduler
echo 1 > /sys/block/$DEVICE/device/queue_depth
echo 4 > /sys/block/$DEVICE/queue/nr_requests

I didn't test each one in isolation so I am not sure if setting every one of them is required, but this combination does work for me.

2
  • Hi, for me echo 3600 > /sys/block/$DEVICE/device/timeout was enough - but this is temporary only, I'd like a persisted solution.
    – mike
    Aug 8, 2020 at 5:24
  • I started by only changing that timeout and that was not enough for me. Changing /sys/block/$DEVICE/device/timeout timeout and /sys/block/$DEVICE/device/eh_timeout may be all that is required, though.
    – gregjhogan
    Aug 9, 2020 at 17:43
0

I have had a good experience using F2FS as opposed to XFS or ext4 on SMR drives. My ext4 exhibited similar behaviour to what you described above on my SMR drives and forced me to investigate SMR solutions in Linux. I also experienced the timeout issues you describe. I am also using Ubuntu, but the later Ubuntu 18.04.3 LTS version.

Firstly, I would never recommend SMR drives for servers that have high random r/w ops. Examples of use-cases where you want to avoid using SMRs would be database and NAS applications with high r/w throughput. My use case is for external backups of a NAS, which is OK because it isn't time critical and also is mostly sequential.

The first thing to do is to obtain the F2FS filesystem, which under 18.04 is as easy as:

sudo apt install f2fs-tools

Using gparted, delete ALL partitions from the SMR drive, then use gparted to create a F2FS partition spanning the entire drive.

My drives (Toshiba) came preformatted with 2 partitions for use on MS-Windows OS machines. If I kept the small first partition then the write speeds were abysmal no matter what filesystem I installed. I strongly suspect the first partition is where the non-SMR portion of the drive is allocated to use for its journal and other metadata. My experience has shown it is very important that the created filesystem has access to this area to get the benefit of it.

Unfortunately, gparted doesn't seem to have a place where I can set up the options to properly create a filesystem suitable for a block zoned SMR drive. After noting the partition identifying information, I exited from gparted and manually ran the mkfs command, this time adding the magic sauce:

sudo mkfs.f2fs -fm /dev/XXXX

where XXXX is your partition you identified earlier in gparted. That -m option is very important as it tells F2FS to use the blocked zone capability of the SMR drive. Without it, my experience shows you will languish in shingle hell.

Having done that (and after mounting it), writing to the drive was fairly consistent. I mostly get write speeds between 117MB/s to 105MB/s. Very occasionally for a few seconds, the write speeds dip to 70-80MB/s.

I suspect that the dip is where the SMR drive has to play catchup by re-writing areas of the drive where there is shingle overlap. Fortunately, that does not happen all that often, although I do admit that I have not (yet) utilised even half the available space on the hard drive. When that happens, I expect that the shingled writes will happen more often and the backups will take longer as a result. However, right now, it is doing a very good job of avoiding shingled areas of the platter and I struggled to find many examples exhibiting a slowdown. It also seems to be utilising the unshingled areas of the device for its metadata (journal), which avoids the massive slowdowns you described.

I have also noticed that the F2FS takes about 10 seconds to flush through the remaining data after the reads have finished and the command prompt has returned. It is important not to unmount the device or unplug it during this time to avoid data loss. If you are using a shell script, please bear that in mind.

I think you will agree that my write rates using F2FS far exceed the ones that you have demonstrated using xfs. Also, I didn't need to change any of the timeouts to make this work.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .