13

I implemented my own Serial-ATA Host-Bus-Adapter (HBA) in VHDL and programmed it onto a FPGA. A FPGA is chip which can be programmed with any digital circuit. It's also equipped with serial transceivers to generate high speed signals for SATA or PCIe.

This SATA controller supports SATA 6 Gb/s line rates and uses ATA-8 DMA-IN/OUT commands to transfer data in up to 32 MiB chunks to and from the device. The design is proven to work at maximum speed (e.g. Samsung SSD 840 Pro -> over 550 MiB/s).

After some tests with several SSD and HDD devices, I bought a new Seagate 6 TB Archive HDD (ST6000AS0002). This HDD reaches up to 190 MiB/s read performance, but only 30 to 40 MiB/s write performance!

So I dug deeper and measured the transmitted frames (yes that's possible with a FPGA design). As far as I can tell, the Seagate HDD is ready to receive the first 32 MiB of a transfer in one piece. This transfer happens at maximum line speed of 580 MiB/s. After that, the HDD stalls the remaining bytes for over 800 ms! Then the HDD is ready to receive the next 32 MiB and stalls again for 800 ms. All in all an 1 GiB transfer needs over 30 seconds, which equals to circa 35 MiB/s.

I assume that this HDD has a 32 MiB write cache, which is flushed in between the burst cycles. Data transfers with less than 32 MiB don't show this behavior.

My controller uses DMA-IN and DMA-OUT command to transfer data. I'm not using the QUEUED-DMA-IN and QUEUED-DMA-OUT command, which are used by NCQ capable AHCI controllers. Inplementing AHCI and NCQ on a FPGA platform is very complex and not needed by my application layer.

I would like to reproduce this scenario on my Linux PC, but the Linux AHCI driver has NCQ enabled by default. I need to disable NCQ, so I found this website describing how to disable NCQ, but it doesn't work.

The Linux PC still reaches 190 MiB/s write performance.

> dd if=/dev/zero of=/dev/sdb bs=32M count=32
1073741824 bytes (1.1 GB) copied, 5.46148 s, 197 MB/s

I think there is a fault in the article from above: Reducing the NCQ queue depth to 1 does not disable NCQ. It just allows the OS the use only one queue. It can still use QUEUED-DMA-** commands for the transfer. I need to realy disable NCQ so the driver issues DMA-IN/OUT commands to the device.

So here are my questions:

  1. How can I disable NCQ?
  2. If NCQ queue depth = 1, is Linux's AHCI driver using QUEUED-DMA-** or DMA-** commands?
  3. How can I check if NCQ is disable, because changing /sys/block/sdX/device/queue_depth is not reported in dmesg?
  • 3
    kernel parameter libata.force=noncq? – frostschutz Jun 23 '15 at 9:32
  • Thanks, this helped a lot, to disable NCQ completely. I also solved the write performance issue. – Paebbels Jun 23 '15 at 12:23
  • 1
    > dd if=/dev/zero of=/dev/sdb bs=32M count=32 Dunno what you intended to do with that; but it will erase both the MBR and gazillions of blocks beyond. Doing this on a drive with the main system running on it (and grub installed on MBR, as in my case) would be fairly dangerous ;) Thought I'd write this here as a comment, to prevent some less-experienced folks from experimenting with your "cool" line...;) – syntaxerror Aug 12 '15 at 17:17
  • @syntaxerror The HDD is connected to an FPGA board. In such an environment it's far to complex to write hardware routines that perfectly handle MBRs and file system acesses. So I'm using the HDD as a raw media on both sides. In the FPGA view it's a big linear memory. In Linux I'm using /dev/sdg and a C programm to read and write the continues data. – Paebbels Aug 12 '15 at 23:02
  • @Paebbels Oops, I should not have overlooked the FPGA bit. Well, this is indeed something completely different to our common HDDs connected to buses of mainboards of desktop PCs or notebooks ;-) "In such an environment it's far to complex to write hardware routines that perfectly handle MBRs and file system acesses". True. You won't be able to do without a HDL. And I can imagine that programming such thing is not for the faint of heart...even though the Wikipedia code example suggests that it's an utter walk in the park ;-) – syntaxerror Aug 13 '15 at 0:21
11

Thanks to @frostschutz, I could measure the write performance in Linux without NCQ feature. The kernel boot parameter libata.force=noncq disabled NCQ completely.

Regarding my Seagate 6TB write performance problem, there was no change in speed. Linux still reaches 180 MiB/s.

But then I had another idea:
The Linux driver does not use transfers of 32 MiB chunks. The kernel buffer is much smaller, especially if NCQ with 32 queues is enabled (32 queues * 32 MiB => 1 GiB AHCI buffer).

So I tested my SATA controller with 256 KiB transfers and voilà, it's possible to reach 185 MiB/s.

So I guess the Seagate ST6000AS0002 firmware is not capable of handling big ATA burst transfers. The ATA standard allows up to 65.536 logical blocks, which equals 32 MiB.

SMR - Shingled Magnetic Recording

Another possibility for the bad write performance could be the shingled magnetic recording technique, which is used by Seagate in these archive devices. Obviously, I triggered a rare effect with my FPGA implementation.

  • 1
    In my experience, disabling NCQ gives a large boost in performance. I have tried this on desktop systems, servers, you name it. Even using 100% "server" high-performance hardware that you would think would benefit from NCQ. Nope, it's all worst than just disabling it. IMHO NCQ is one of the worst things to happen to hard-drives. I have never once seen it provide a benefit under any circumstances, be it dedicated RAID card or on-board chipset. – CR. Dec 31 '17 at 22:25
  • Have you done burst operations or random access? NCQ has no impact on burst operations but improves random access. – Paebbels Jan 1 '18 at 1:15
  • Sorry, but you haven't answered my question. Another question is, what hardware have you used a simple consumer mainboard or a workstation/server mainboard or dedicated RAID controller. Many implementations do not support as many outstanding requests as designed by NCQ / AHCI. – Paebbels Jan 2 '18 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.