IO scheduler has a queue depth of 128 by default. And block device has a queue depth of 32 by default. Where does the IOs are buffered. Does IO scheduler send all IOs to scsi low level drivers and driver buffer the IOs. Or there is throttling somewhere else.
1 Answer
IO schedulers can be "non work-conserving". That is, they can choose not to send down IOs immediately. There is a IO scheduler technique known as "idling". It is particularly useful for spinning disks (i.e. not SSDs). Both CFQ and BFQ schedulers use idling.
Also remember that write() calls can be buffered in the page cache, before it is written back as an IO request. And underneath the block device queue, there may be a device write cache. It's all very complex and leads to various strange behaviours :-).
Note not all block devices have a default queue depth of 32, that's just the common queue depth for SATA NCQ. SCSI TCQ devices can allow more, at least the standard does. (And NVMe devices, can definitely allow more, and can have multiple parallel queues as well).