3

I use a Measurement Computing DAQ in Ubuntu to perform continuous analog reads and writes from another system connected to the board. I have been using Ubuntu 16.04 (which went up to Linux kernel 4.15) for about five years now. I was recently exploring upgrading the system to Ubuntu 20.04 - 22.04 and each of those operating systems ships with Linux kernel 5.10 - 5.15. I am noticing that I am getting what appears to be periodic interrupts that are quite drastic (about 50 milliseconds) for every kernel 5.10 or higher. So something appears to have changed from the 5.9 kernel to the 5.10 kernel that is affecting system read() and write() calls with the A/D board. The differences can be seen in my data acquisition software:

enter image description here

And also in an average loop time program I have (that loops through successive read and write calls - along with some math in between):

enter image description here

Note how the maximum times I am seeing go from about 43 microseconds for Linux kernel 5.9 and below to 50 milliseconds for Linux kernel 5.10 and above. I obviously would like to fix this problem, but I am not sure what was changed that could have caused it. Does anyone have any idea what the culprit is, and if it could be fixed by perhaps changing a kernel parameter in the GRUB bootloader? Any pointers at all would certainly be appreciated. Thanks!

EDIT:

I have implemented a minimal example where we continuously call write system commands to update the DAC Outputs.

At minimum, the DAC Write command is calling "get_user" to obtain data from user-space to kernel, and calling "outw" to write the data into the DAC Register.

Now when we are executing the minimal example, we're doing back-to-back write system commands and we're noticing this 50 millisecond delay.

However, when we add a 1 microsecond delay between the write system commands, then the 50 millisecond delay vanishes. Is this possibly an issue with trying to access the user-space information or writing from the kernel to the device too quickly?

Is there a way to analyze what the kernel is doing between accessing user-space and writing data from the kernel to device?

Improve this question
8
  • how is it board connected: serial port? pci-e? what about "sys" cpu usage: is it high?
    – user996142
    May 13, 2022 at 19:17
  • The board is connected via pci-e. Cpu usage isn't high. The processes are confined to run on individual cores and there are eight of them. It is definitely kernel related.
    – Leigh K
    May 13, 2022 at 19:25
  • Does it use kernel module/driver? Which one? I asked about cpu usage because my idea was an interrupt problem.
    – user996142
    May 13, 2022 at 19:32
  • 1
    git bisect to the rescue. There are thousands of changes, no one has any clue what exactly affects you, not even kernel developers.
    – Artem S. Tashkinov
    May 13, 2022 at 20:35
  • 2
    Although I also think this is going to need git bisect to fully figure out, kernelnewbies.org maintains "human-readable kernel changelogs" for each Linus's kernel release.. In this case, the page for the kernel version 5.10 might be useful for an overview. On the other hand, the pci-das1602-16 driver seems to have had essentially no changes since November 2016 at least, and the newest kernel mentioned in its README is 3.10.11, so that driver might not exactly be up to the latest kernel coding practices.
    – telcoM
    May 13, 2022 at 22:50

3 Answers 3

Reset to default
3

I was having the same kind of problem with real-time threads when updating the kernel on my computer. For me sysctl -w kernel.sched_rt_runtime_us=-1 helped.

The change that broke this was: Disable RT_RUNTIME_SHARE by default

Improve this answer
2
  • This resolved the issue. Many thanks! I wonder if this kernel parameter can be set via the grub configuration file in Ubuntu?
    – Leigh K
    Jun 29, 2022 at 20:19
  • Can't do this in grub, but you can add the parameter to /etc/sysctl.conf, run sysctl -p, then reboot to have the parameter load at boot time rather than run time. Thanks again.
    – Leigh K
    Jun 30, 2022 at 15:06
1

5.9 -> 5.10 differences can be seen here

Various scheduler updates introduced in 5.10 can be seen here

Improve this answer
0

It appears that something changed regarding priority scheduling between these two Linux kernels. The program that is running requests a scheduling priority of 90, and that is what is causing the interrupts. Removing the priority scheduling (or setting it's priority to zero) removes the interrupts. I have yet to determine what exactly changed and what a solution would look like that still allows priority scheduling, but the root cause of the issue seems to be determined.

Improve this answer
4
  • sched prio of 90 ? You mean you are running this app SCHED_FIFO or SCHED_RR ? You wrote (OP) when you add the 1 µs delay between 2 writes the delay vanishes ; How did you program that ? (nanosleep or equiv or something else)
    – MC68020
    Jun 21, 2022 at 18:03
  • @MC68020 This is running SCHED_FIFO. The 1 µs delay was done using udelay().
    – Leigh K
    Jun 21, 2022 at 19:09
  • Do you get other SCHED_FIFO or SCHED_RR tasks running ? In particular threaded interrupts ? what are the priority of these kernel threads ? BTW, why did you use udelay since I presume you are not in atomic context. Can you retry using usleep ? (My first guest would be with the HRTIMERS. There has actually been several related patches commited to 5.10, in particular lore.kernel.org/all/… still trying to figure out how these could impact though)
    – MC68020
    Jun 21, 2022 at 20:07
  • As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.
    – Community Bot
    Jun 28, 2022 at 15:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .