I have a process that reads data from a hardware device using DMA transfers at a speed of ~4 * 50MB/s and at the same time the data is processed, compressed and written to a 4TB memory mapped file.
Each DMA transfer should (and do on average) take less than 20ms. However, a few times every 5 minutes the DMA transfers can take up to 300ms which is a huge issue.
We believe this might be related to the kernel flushing dirty memory mapped pages to disk. Since if we stop writing to the mapped memory the DMA transfers durations are just fine. However, we are confused as to how/why this could affect the DMA transfers and whether there is a way to avoid this?
The hardware device has some memory to buffer data but when the DMA transfers are this slow we are loosing data.
Currently we’re doing testing on Arch Linux with a 4.1.10 lts kernel, but we’ve also tried Ubuntu 14.04 with mostly worse results. Hardware is a HP z820 workstation, 32GB RAM and dual Xeon E5-2637 @ 3.50Ghz (http://www8.hp.com/h20195/v2/GetPDF.aspx/c04111177.pdf).
We also tried a Windows version of our software as well that does not suffer from this specific issue but has lots of other issues.