We run some software on various systems at
realtime priority (
Priority: 99) To hit certain frame requirements in the software our engineers work on.
one of our side projects doesn't "really" need the same realtime performance, but they need some functionality that is only achievable when running in realtime. IE frame misses aren't a critical failure state, but they need to try and hit in order for certain other things to happen properly. It's a confusing mess.
Anyway, While running the software realtime, they also use the system for other quick tasks, often opening terminal windows, running a quick program, then Ctrl+d to drop the terminal, and move on with their life. Doing this leaves copies of the applications in an un-interrupt able sleep, that over time eats up system resources and locks up the system. If the software is stopped, it immediately resolves all of those sleeping processes and finishes closing everything.
My thoughts are that whatever cleanup process runs to finish cleaning up those processes simply wont complete with the system holding realtime priority over things. I realize "this isn't how you should use a realtime system" and all of that, but unfortunately its an issue that they refuse to concede the point on, and I have to find a magical solution to a simple problem(don't use a realtime system as a workstation.)
So my question is: Is there a way of manually forcing the cleanup process from time to time to take precedence over the maximum running process?
I've considered trying to get fancy with
cgroups but that seems like a really complicated solution to the problem, and one that I'm not sure will actually work even if i take the time to learn how to work well with
cgroups and the various kernel timing stats