1

I have a process that reads data from a hardware device using DMA transfers at a speed of ~4 * 50MB/s and at the same time the data is processed, compressed and written to a 4TB memory mapped file.

Each DMA transfer should (and do on average) take less than 20ms. However, a few times every 5 minutes the DMA transfers can take up to 300ms which is a huge issue.

We believe this might be related to the kernel flushing dirty memory mapped pages to disk. Since if we stop writing to the mapped memory the DMA transfers durations are just fine. However, we are confused as to how/why this could affect the DMA transfers and whether there is a way to avoid this?

The hardware device has some memory to buffer data but when the DMA transfers are this slow we are loosing data.

Currently we’re doing testing on Arch Linux with a 4.1.10 lts kernel, but we’ve also tried Ubuntu 14.04 with mostly worse results. Hardware is a HP z820 workstation, 32GB RAM and dual Xeon E5-2637 @ 3.50Ghz (http://www8.hp.com/h20195/v2/GetPDF.aspx/c04111177.pdf).

We also tried a Windows version of our software as well that does not suffer from this specific issue but has lots of other issues.

  • 1
    On what hardware do you run? – Dmitry Grigoryev Oct 26 '15 at 9:17
  • 2
    I suppose your disks use DMA transfers as well. You should check the manual to see if you can configure the memory controller and what are the options. – Dmitry Grigoryev Oct 26 '15 at 10:10
  • 1
    @AndrewHenle: It's writing continuously and sequentially and overwriting the oldest data. The data will be read about 10 seconds after it's been written so we mostly need about 10 seconds of file cache to avoid read io which would totally bust disk performance as you hint. There will be very little random IO. Both read and write will be sequential except from some very unusual binary seeking. Note, that for this test we are not reading at all from the file since we haven't even gotten just writing working yet. – ronag Oct 26 '15 at 10:24
  • 1
    @DmitryGrigoryev: Replacing Intel RAID with Linux software RAID seems to improve things we now have a intermittent max of 100ms, still not a full solution though. Is there any way in Linux to get a bit more information what the DMA controller is doing? – ronag Oct 26 '15 at 11:10
  • 1
    You can try chrt -f and ionice -c 1 to give realtime scheduling to your process, may help with io queueing. Also, stop cron and other background jobs you are not interested in (eg temporarily with kill -stop then kill -cont). – meuh Oct 26 '15 at 14:30
2

Linux has some realtime options, though it is not a realtime kernel as such. This allows a process to demand that it is scheduled before non-realtime processes, as soon as it is ready, and to hold on to the cpu for as long as necessary.

By default processes are given scheduling policy SCHED_OTHER. You can set this to realtime SCHED_FIFO for a given running pid with chrt -f -p prio pid, or prefix the command with chrt -f prio when you start it. The prio priority is independent of normal processes, and is only used when realtime processes compete for resources. ps shows these priorities as negative values (eg -21 for realtime prio 20).

ionice --class 1 -p pid can also help scheduling your process with preferential realtime io queueing.

1

Does your device operate continously for minutes or are there any regular pauses in the transfers?

If there are pauses, you can force the kernel to empty buffers and cache during those, so this activity wouldn't interfere with the DMA transfers. Alternatively, you could configure your kernel to have BDFLUSHR interval of 1 second, so that the kernel has less data to write each time it decides to flush buffers.

If you need to insure continuous operation, you will need RAM with more channels, so that CPU and your device can access the memory simultaneously (as it turns out, you already have a 4-channel memory controller). Make sure you configure your RAM in unganged mode, if this option is available. Make sure you have installed similar DRAM modules in 4 slots corresponding to memory channels, so that your memory controller can in fact operate in 4-channel mode.

  • The device runs continuously in 40 ms intervals (it's a 4x video capture device). – ronag Oct 26 '15 at 9:45
  • So you have about 20ms of DMA transfer + 20ms of pause? – Dmitry Grigoryev Oct 26 '15 at 9:47
  • I don't know the exact values but yes, < 20ms DMA transfer and then pause. – ronag Oct 26 '15 at 9:49
  • Is ganged vs unganged a BIOS setting? – ronag Oct 26 '15 at 9:52
  • Yes, if this is configurable. Sometimes the mode is just hardcoded in the chipset. – Dmitry Grigoryev Oct 26 '15 at 9:56
0

I'd guess that you haven't modified kernel dirty pages settings. For your use case, I would try something like this:

/proc/sys/vm/dirty_background_bytes:50000000
/proc/sys/vm/dirty_bytes:4000000000
/proc/sys/vm/dirty_expire_centisecs:100
/proc/sys/vm/dirty_writeback_centisecs:20

(See https://www.kernel.org/doc/Documentation/sysctl/vm.txt for details.)

The problem is basically that the default kernel limits are problematic if you system has lots of RAM and slow enough storage device and you're seeking low worst case latency. In practice, system IO subsystem buffer is filled up and it needs to force writing processes to sleep until enough data has been written to block devices ("flushing dirty pages").

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.