3

I have a multipath device I'm interested in:

[root@xxx dm-7]# multipath -ll mpathf
mpathf (3600601609f013300227e5b5b3790e411) dm-7 DGC,VRAID
size=3.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| |- 7:0:1:5 sdl 8:176  active ready running
| `- 8:0:1:5 sdx 65:112 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 7:0:0:5 sdf 8:80   active ready running
  `- 8:0:0:5 sdr 65:16  active ready running

So it looks like the block devices backing this path are sdf, sdr, sdl, and sdx. Just taking sdf as an example I've set it's I/O scheduler as being noop:

[root@xxx dm-7]# cat /sys/block/sdf/queue/scheduler
[noop] anticipatory deadline cfq

The mpathf device maps to /dev/dm-7 for the actual block device. I've just noticed that this has an I/O scheduler as well:

[root@xxx dm-7]# cat /sys/block/dm-7/queue/scheduler
noop anticipatory deadline [cfq]

Question: which one takes precedence? The scheduler on the multipath device or on the device it ends up relaying the I/O through?

I'm of course assuming that IOPs aren't scheduled twice (once for the mpath device and another for the individual block device the I/O is redirected into).

2

Short Answer:

Device mapper in kernel versions after 2.6.31 (released September 9th 2009) includes support for "request-based" dm targets. So far only the only request-based dm target is dm-multipath.

For the targets that remain BIO (i.e everything except multipath) scheduler selection is still present but irrelevant as the DM target hands off the IOP prior to that point.

For request-based targets the scheduler selection supersedes what is set on the individual block device as multipathd will be communicating the requests directly to the underlying SCSI device (/dev/sg4, /dev/sg5, etc).

Additional information:

User application I/O is referred to as BIO (block I/O). BIO is sent to the scheduler/elevator for request merging/ordering and then is sent as a "request" to the lower level devices.

Historically, dm-multipath has been solely at the BIO level. This created a problem where traffic from separate BIO's would be merged by the block device (sdb, sdf, etc) resulting in some request queues being shorter/less used than other possible paths. BIO dm-multipath was also unable to have visibility on things like retry events or the like, as it was hidden by the block device (/dev/sda, /dev/sdb, etc).

The sysfs object for multipath block device prior to the change (RHEL 5):

[root@xxxsat01 dm-1]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.10 (Tikanga)
[root@xxxsat01 ~]# uname -r
2.6.18-371.8.1.el5
[root@xxxsat01 dm-1]# cat dev
253:1
[root@xxxsat01 dm-1]# ll
total 0
-r--r--r-- 1 root root 4096 Jan 29 13:54 dev
drwxr-xr-x 2 root root    0 Apr 29  2014 holders
-r--r--r-- 1 root root 4096 Jan 29 13:54 range
-r--r--r-- 1 root root 4096 Jan 29 13:54 removable
-r--r--r-- 1 root root 4096 Jan 29 13:54 size
drwxr-xr-x 2 root root    0 Jan 25 06:25 slaves
-r--r--r-- 1 root root 4096 Jan 29 13:54 stat
lrwxrwxrwx 1 root root    0 Jan 29 13:54 subsystem -> ../../block
--w------- 1 root root 4096 Jan 29 13:54 uevent

Post-Change (RHEL 6):

[root@xxxlin01 dm-1]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)
[root@xxxlin01 ~]# uname -r
2.6.32-431.3.1.el6.x86_64
[root@xxxlin01 dm-1]# cat dev
253:1
[root@xxxlin01 dm-1]# ll
total 0
-r--r--r-- 1 root root 4096 Jan 29 13:58 alignment_offset
lrwxrwxrwx 1 root root    0 Jan 29 13:58 bdi -> ../../bdi/253:1
-r--r--r-- 1 root root 4096 Jan 29 13:58 capability
-r--r--r-- 1 root root 4096 Jan 29 13:58 dev
-r--r--r-- 1 root root 4096 Jan 29 13:58 discard_alignment
drwxr-xr-x 2 root root    0 Jan 29 13:58 dm
-r--r--r-- 1 root root 4096 Jan 29 13:58 ext_range
drwxr-xr-x 2 root root    0 Jan 29 13:58 holders
-r--r--r-- 1 root root 4096 Jan 29 13:58 inflight
drwxr-xr-x 2 root root    0 Jan 29 13:58 power
drwxr-xr-x 2 root root    0 Jan 29 13:58 queue
-r--r--r-- 1 root root 4096 Jan 29 13:58 range
-r--r--r-- 1 root root 4096 Jan 29 13:58 removable
-r--r--r-- 1 root root 4096 Jan 29 13:58 ro
-r--r--r-- 1 root root 4096 Jan 29 13:58 size
drwxr-xr-x 2 root root    0 Jan 29 13:58 slaves
-r--r--r-- 1 root root 4096 Jan 29 13:58 stat
lrwxrwxrwx 1 root root    0 Jan 29 13:58 subsystem -> ../../../../class/block
drwxr-xr-x 2 root root    0 Jan 29 13:58 trace
-rw-r--r-- 1 root root 4096 Jan 29 13:58 uevent

Since the kernel is unaware of what individual targets do it presents the same sysfs attributes regardless of what type of device mapper device it is. Since the request is then relayed to the block-level devices, the device mapper's scheduler is never invoked and so this setting is essentially a noop with other dm targets.

Further Reading:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.