I have a machine running Debian 8 (jessie) downgraded to kernel 3.2-rt. This is running an application which is set to realtime priority (SCHED_RR), wakes up every 1ms to do some processing, and every 100ms sends a serial message to another PC. This works perfectly. (The machine doesn't really run anything else other than the default services; in particular it's not a desktop machine.)
When trying to upgrade this to Debian 9 (stretch) with kernel 4.9-rt I have encountered considerable problems; one of the biggest issues is clock stability. While initially it seems to run ok, a short time after boot I get this:
clocksource: timekeeping watchdog on CPU0: Marking clocksource 'tsc' as unstable because the skew is too large:
clocksource: 'acpi_pm' wd_now: 74c6dd wd_last: 8dd916 mask: ffffff
clocksource: 'tsc' cs_now: 119ac91c7b1 cs_last: 1158a25441d mask: ffffffffffffffff
clocksource: Switched to clocksource acpi_pm
The first time this occurred, the above message reported switching to "hpet"; after this switch occurred all the timing on the machine appeared to be wonky -- the code uses clock_gettime(CLOCK_MONOTONIC, ...) for timing and this was reporting that it was now waking up every 4ms instead of every 1ms, the serial message was actually being transmitted every 200ms instead of every 100ms, but the timestamps inside the serial message indicated that it thought it was sending them every 66ms. (It's possible that some of these were counting wakeup ticks rather than actual ms.)
I tried disabling HPET in the BIOS, which led to the error shown above, and again it runs ok on boot until logging the above at which point while it mostly seems to run ok, sometimes some of the timers report negative durations.
The other main thing going wrong is that the processing code that runs every 1ms takes about 200us to run in Jessie but 900us to run in Stretch, for no readily apparent reason. I've tried running perf record over it and it reports that most of the time (40% of samples) is being spent in sys_clock_gettime.
This is the same hardware in both cases: a Supermicro X11SSQ motherboard with an i7-7700K CPU in AMD64 architecture. CPU throttling is disabled in the BIOS and it does report constant_tsc and nonstop_tsc.
Possibly of interest is that the application was doing a busy-wait loop (if clock_gettime() < wakeup then sched_yield(), in simplified pseudocode) to schedule the processing code.
As an experiment I've changed it to sleep instead (clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &wakeup)) and at least thus far the clocksource error hasn't reoccurred and it has remained using the tsc clock, which seems to work correctly (unlike the other clocks).
However the wakeup jitter is now 10x worse: 700-1200us (SD 36us) for sleep vs. 890-1120us (SD 2us) for busy-wait.
As a further experiment I tried changing it to SCHED_DEADLINE (with a single sched_yield()), but this has the same behaviour as clock_nanosleep.
Any idea why this behaviour changed between Jessie and Stretch (or between 3.2 and 4.9, more likely)? Is there a better way to accomplish a low-jitter wait without apparently breaking the system clock?
