17

We are running a real-time process on a non real-time kernel (CentOS 6), and this is probably not going to change.

We have a streaming video application that requires about 500 MB/s of PCIe traffic from a custom FPGA continuously for 1.5 hours at a time. The application works pretty well - most of the time. However, we've had situations where it appears that the kernel just stops responding to servicing PCIe or memory requests for up to 500 milliseconds at a time. This appears to happen during bursty file IO from another thread. I've found it impossible to try to replicate this problem by just doing lots of dummy file IO from user space while the main application is running.

Is there a way to force (simulate) a global "freeze" of the Linux kernel (in particular, stopping PCIe or all DDR3 memory accesses or something like that) so that we can reproduce this problem?

We have buffering up to 10 milliseconds implemented right now into internal FPGA memory, but that is not enough. We can buffer to FPGA DDR3 and then dump to the host, but we need a method to test this new feature under duress.

We don't want the kernel to freeze or lock up permanently. We would like the ability to set the time interval.

I am looking for something along the lines of writing magic values to /proc/sys/vm temporarily that makes the system virtually crawl, and then reverting back after a few hundred milliseconds, but looking at the number of possible ways to break it is not for a newbie like me (https://www.kernel.org/doc/Documentation/sysctl/vm.txt) . Maybe some numactl magic?

Improve this question
6
  • My hunch is that this requires writing a kernel module. You'll need to freeze all threads on all CPUs somehow, and arrange to restart on a timer interrupt.
    – Gilles 'SO- stop being evil'
    Oct 9, 2014 at 22:20
  • I don't want to freeze the threads, I want to freeze the kernel! I mean, I want to prevent access to hardware (memory and/or PCIe and/or disk) for a short time. If that doesn't work, I don't mind making things very unoptimized, disabling L1 cache, etc. I just don't know how to do this.
    – Mark Lakata
    Oct 10, 2014 at 0:45
  • 1
    Ah, so you don't want to freeze the kernel, you only want to freeze the part of the kernel that responds to some hardware? That, too, would require diving pretty deep into the kernel.
    – Gilles 'SO- stop being evil'
    Oct 10, 2014 at 0:52
  • I don't mind freezing the kernel completely, as long as the hardware is frozen as part of it.
    – Mark Lakata
    Oct 20, 2014 at 16:54
  • 1
    It turns out, the issue is related to TLB thrashing as the host CPU flushes some IO buffers (we are using HDF5 to write files), and this TLB thrashing is causing the coprocessor to also thrash, as it is a NUMA system. I guess all we need now is a reliable way of programmatically causing TLB thrashing for a controlled amount of time.
    – Mark Lakata
    Jan 5, 2015 at 19:14

4 Answers 4

Reset to default
9
+100

One option to do a quick test could be to use a KGDB enabled kernel and stop the kernel manually and test, see this link.

On another note, things I remember that could cause your pauses:

  • cpufreq, cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_transition_latency, the value is in ns (4000 in my AMD FX(tm)-8120 Eight-Core Processor) shouldn't be a problem, but check
  • Thermal throttling either the cpu itself or the voltage regulator module.
  • NAPI and/or heavy network traffic
  • PCIe ASPM (cat /sys/module/pcie_aspm/parameters/policy)
  • Contention in the buffers of your destination device (hard disk, nic...)
  • Bug in the firmware of some device in the PCIe bus (even if you are not using it), you can try powering them off with /sys/bus/pci/devices/$DEVICE/power/control
Improve this answer
4
  • Could I use kdb instead of kgdb to do the same? I've never used either. Is this like the "Stop-A" command sequence on Sun workstations of yesteryear? If I just do a quick SysRq-g, then type "go", will I have a high probability of not breaking the system? (ref: kernel.org/pub/linux/kernel/people/jwessel/kdb/…)
    – Mark Lakata
    Oct 15, 2014 at 23:58
  • 1
    Probably you'll be able to use kdb. Be aware that it should work with usb connected keyboards, but try to have a PS/2 one handy just in case. And this is a very low level debugger (kernel land), so as always, keep backups and if it breaks you get to keep both pieces :).
    – Jorge Nerín
    Oct 17, 2014 at 8:36
  • Before resorting to tweaking with the kernel I would first try to unload unused kernel modules for PCIe devices that could be using the bus (graphics drivers most notably), and either physically removing devices from the system or powering them off. PCIe 1.0 x1 has a bandwidth of 250MB/s and PCIe 2.0 x1 goes up to 500MB/s, are both the origin and destination device free to accept such sustained rate without interruptions or do they have more lanes to allow more headroom?
    – Jorge Nerín
    Oct 17, 2014 at 8:59
  • Another possible source of the lag might be some ACPI power management handler of some device or maybe even some SMM CPU handler waiting for an external event.
    – Franki
    Nov 27, 2014 at 5:28
2

Can we have more details on how your application is communicating with the FPGA ? Is it the application that reads the buffer from the FPGA, or the FPGA that sends interrupt to the kernel (like network cards) ?

I expect it to open a block/char in /dev and then communicate with it. This means it uses a driver to do the communication between the application and the /dev/XXX file.

I'd like to have the output of : cat /proc/interrupts ; lsmod ; ls -al /dev/yourmod

Here's the ideas :

  • If it's interrupt driven, you can set the CPUs PIC to disable the corresponding IRQ, then re-enable it. This will cause every request of the card to be ignored (without the card to be aware of it).
  • if it's like a buffer read, you may:
    • Put your application in sleep state, so the data from the FPGA won't be read, and your buffer will fill, then wake up your application and continue the read.
    • Use "crash" or "kgdb" to change the "read" value to a "noop" for a few secs, then set it back to the default function.

Please provide all information you may find to be useful.

Improve this answer
2
  • FPGA does DMA writes to host memory, and during these outage periods, the FPGA is not able to write to host memory, so its internal FIFO backs-up. There is a message-based interface to the host process (happens over PCIe), but I am certain that this is not involved. For validation purposes, I basically need a way of prohibiting the FPGA hardware to write to host memory for a few hundred milliseconds. I don't want to solve the memory problem, but I want to make sure that our implementation on the FPGA is able to deal with a memory outage (up to 1000 ms).
    – Mark Lakata
    Oct 20, 2014 at 18:39
  • Ok, if it's using DMA, you may have a look on : kernel.org/doc/Documentation/DMA-ISA-LPC.txt particulary on the claim_dma_lock() and dma_disable(). However, you'll need to know the addresses used by your FPGA.
    – Adrien M.
    Oct 20, 2014 at 19:50
1

Not sure if it helps. But if you can write a kernel module that calls the suspend function of another device's kernel module, that may do.

Each PCI device can be suspended according to the header file http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/include/linux/pci.h#L479

For example, here's Intel e1000 NIC's suspend function http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/drivers/net/e1000e/netdev.c#L4643

From what I can recall, this function was mainly used when the system goes to hibernation, device driver need to save the current running status and turn itself off.

Improve this answer
1
  • thanks, but I don't think that will work. I don't really want to suspend a device, which is the kernel telling the device to prepare for hibernation; I want the the kernel to ignore the specific device (in this case the FPGA daughter board) without it knowing (other than long latencies or timeouts) -- or I want to stop all SDRAM memory transfers.
    – Mark Lakata
    Oct 14, 2014 at 22:12
0

I think you are thinking along the wrong lines. Your goal is clear.

The way is not to stop the rest of the processes but to give your main processes near real-time sceduling priority. Use nice for your important user-space-processes for that.

The more difficult problem is the PCIe interrupt handling, which resides in kernel-space.

Since hardware is involved, you should start taking a closer look at the involved PCIe lane on your mainboard and how that is possibly connected to a specific CPU socket.

irqbalance does normally a good job here, but you might configure its bahaviour to suit your needs.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .