I'm resyncing a RAID1 array. /proc/mdstat shows it progressing at expected speeds, but iotop shows the disks as idle in all respects. It does not show the IO, which I would guess is being performed by the kernel thread md127_resync, despite man iotop mentioning kernel threads.

How can we explain this IO not showing up in iotop? Does the explanation suggest other types of IO that would also not show up?

I am using Debian 9.


There is a hint in a close reading of man iotop

At least the CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING, CONFIG_TASKSTATS and CONFIG_VM_EVENT_COUNTERS options need to be enabled in your Linux kernel build configuration.

iotop displays columns for the I/O bandwidth read and written by each process/thread during the sampling period. It also displays the per‐ centage of time the thread/process spent while swapping in and while waiting on I/O. For each process, its I/O priority (class/level) is shown.

In addition, the total I/O bandwidth read and written during the sampling period is displayed at the top of the interface. Total DISK READ and Total DISK WRITE values represent total read and write bandwidth between processes and kernel threads on the one side and kernel block device subsystem on the other. While Actual DISK READ and Actual DISK WRITE values represent corresponding bandwidths for actual disk I/O between kernel block device subsystem and underlying hardware (HDD, SSD, etc.). Thus Total and Actual values may not be equal at any given moment of time due to data caching and I/O operations reordering that take place inside Linux kernel.

CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING, CONFIG_TASKSTATS sound like the per-task statistics that make up Total. Whereas CONFIG_VM_EVENT_COUNTERS is a single overall statistic, i.e. Actual (in /proc/vmstat, if you look up the config option). The VM part, meaning Virtual Memory, strongly suggests this is about kernel buffered IO, i.e. that which goes through the page cache. In other words those statistics are being captured as IO goes from the VM subsystem to the block device subsystem.

It suggests to me that IO which bypasses the VM, will not be counted in iotop. It likely does not include O_DIRECT, for example. This also explains how iotop avoids any double-counting of IO passed through software RAID (and LVM, etc), and why it has no feature to break out statistics for different block devices.

RAID resync IO doesn't need to go through the page cache. It will be using in-kernel APIs, which I think share some similarities with async IO (Linux only supports O_DIRECT for async IO). So it does not suffer the performance limitation of using read() and write() without the asynchronous writeback and readahead provided by the page cache.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.