2

My new NAS has 5 disks. They are all same model.

sde is connected to the controller on the motherboard. sda-sdd is on the raid controller.

Run 'pv /dev/sd[cde]' in parallel:

Device       rkB/s     wkB/s f_await  aqu-sz  %util
sdc      161536.00      0.00    0.00    1.50  99.60
sdd      175104.00      0.00    0.00    1.41  98.80
sde      170880.00      0.00    0.00    1.66 100.00

This is what I expect.

Run 'pv /dev/sd[a-e]' in parallel:

Device       rkB/s     wkB/s f_await  aqu-sz  %util
sda      147456.00      0.00    0.00    1.15 100.00
sdb      142848.00      0.00    0.00    1.74 100.00
sdc      147840.00      0.00    0.00    1.13  99.60
sdd      149120.00      0.00    0.00    1.15  99.60
sde      107008.00      0.00    0.00    1.34  96.40

That the max goes from 175 MB/s to 150 MB/s could be caused by them sharing a bus, and this bus has a maximal total bandwidth.

But note how sde is 30% slower.

Renice 'pv' to 19 for sda-sdd (keep 'pv' for sde at 0):

Device       rkB/s     wkB/s f_await  aqu-sz  %util
sda      137856.00      0.00    0.00    1.04  98.00
sdb      140032.00      0.00    0.00    1.06  99.20
sdc      132480.00      0.00    0.00    1.00  98.80
sdd      132608.00      0.00    0.00    1.02  97.60
sde      140672.00      0.00    0.00    1.73 100.00

Note how sde is now on-par with the rest. This is what I had expected in the normal case (without renicing).

On an idle system I get 80 seeks/sec per drive (as expected):

# parallel --tag -j0 -k --ll seekmaniac ::: /dev/sd[a-e]
/dev/sda        / 81 seeks per second           
/dev/sdb        / 81 seeks per second           
/dev/sdc        / 80 seeks per second           
/dev/sdd        / 81 seeks per second           
/dev/sde        / 82 seeks per second           

When 'pv' is running (as above):

# parallel --tag -j0 -k --ll seekmaniac ::: /dev/sd[a-e]
/dev/sda        o 15 seeks per second           
/dev/sdb        o 13 seeks per second           
/dev/sdc        o 15 seeks per second           
/dev/sdd        o 19 seeks per second           
/dev/sde        o 64 seeks per second           

Note how sde delivers more seeks than the rest.

I assume this behaviour is due to sde is on a different controller.

But how can a controller have that effect?

What explains this behaviour?

Edit

Searching for differences I found (sda=sdb=sdc=sdd):

# diff <(cd /sys/block/sde/; grep . queue/*) <(cd /sys/block/sda/; grep . queue/*) 
19c19
< queue/max_segments:64
---
> queue/max_segments:168
23c23
< queue/nr_requests:2
---
> queue/nr_requests:64
34c34
< queue/write_cache:write through
---
> queue/write_cache:write back

The write cache is unlikely to be relevant when all I do is read.

5
  • hmmm do both controllers (and their drivers) support NCQ? (Natural command queuing would increase overall throughput but has the potential to postpone time-consuming small-data requests including seeks) Nov 27, 2023 at 17:31
  • @MarcusMüller How do I determine if they support NCQ? Also does this fit with the first measurement: sde's controller clearly can deliver 170 MB/s.
    – Ole Tange
    Nov 28, 2023 at 1:30
  • I would have scoured sysfs long enough to find what you have found :) regarding reduction of rate of sde in a high-overall-throughput scenario: Hmmmm interrupts? maybe watch cat /proc/interrupts while you do the first two measurements? Nov 28, 2023 at 12:15
  • sde is connected to the controller on the motherboard Motherboard-based controllers tend to be low-performing commodity hardware. Normally, anyone who needs more performant IO will use an HBA or RAID controller of some sort, so motherboard disk controllers are a place where manufacturers save as much money as they can. Nov 28, 2023 at 18:58
  • 1
    @AndrewHenle He shows that the base performance seems to be similar to all disks, and I guess that the question is why does sde become slower when reading from all 5 disks at once (when even when reading from 3 disks the performance isn't impacted).
    – aviro
    Nov 29, 2023 at 8:33

1 Answer 1

0

This is simply a guess. I hope that some can either prove or disprove this guess.

sde controller has max_segments=64 and nr_requests=2.

sda controller has max_segments=168 and nr_requests=64.

Let us assume it is more work for the CPU to get data from sde: If the queue capacity is smaller, the CPU will have to empty the queue constantly, whereas queue with a bigger capacity will only have to be emptied periodically.

If the CPU misses emptying the queue, it will be full, and the disk will stall. This will on average happen more often if the capacity of the queue is small.

The CPU in the NAS is a low end, slow CPU.

Maybe this could explain the 30% difference in read performance.

Can the difference in seeks also be explained by similar reasoning?

If the queue capacity is low, the CPU will quicker discover that a new seek can be dispatched. If the capacity is high, the CPU will first have to empty the queue before it discovers that a new seek can be dispatched.

Again: I am clearly in deep water here. I honestly do not know if this is what is going on.

1
  • Notice that nr_requests is RW, which means you can change the number. Try to increase it to 64 and check if it changes the results.
    – aviro
    Nov 29, 2023 at 15:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .