I am using a pselect system call in user-space code to wait on a GPIO pin to toggle. I know that it should take no more than 5 milliseconds for this toggle to occur, and I've verified this timing using an oscope. However the pselect call confusingly sometimes takes up to 20 milliseconds to return.
I modified the GPIO driver, the pselect implementation in the kernel, and the pselect implementation in the glibc library, to all print the current time when they are hit. Using these I've verified that the GPIO driver and kernel's pselect implementation are hit in a timely manner, however there is between 6 and 15 milliseconds between when the kernel pselect call returns and the line immediately following glibc's system call into the kernel.
My understanding of how system calls are implemented in the c library is that the architecture-specific macro "INLINE_SYSCALL" switches the CPU between user and kernel modes, passes parameters to the kernel, actually calls the kernel function, and then handles setting errno if applicable. In ARM this logic appears to be a couple of assembly instructions which I would expect to take practically no time at all.
Hardware and Software:
I'm using kernel version 4.1.33 (with the preempt real time patch), and glibc version 2.23, running on ARM cortex A9.
What is happening between when the kernel function returns and the line immediately following the glibc's call into the kernel that could explain any significant loss of time?
Are there any additional tests that I could run to further narrow down the cause of this time loss?
As a sanity check I temporarily replaced the pselect logic with a tight busy loop which requested the value of the gpio pin as fast as it possibly could. This loop only exited when the value of the gpio pin was observed to change, and it always exited within 5 milliseconds, indicating that pselect was in-fact the problem.
After some additional testing I discovered that despite the poll system call doing essentially the same thing as pselect, it does not appear to be subject to the loss of time observed when using pselect. As a result, I am able to work around this issue, however I'd still like to understand its cause, if possible...