I am curious about how data is copied back and forth between CPU and GPU memories when I run a CUDA program. Specifically, I want to know how the Linux kernel is involved in this process.
I had a few possible assumptions:
- The CUDA user library sees the GPU as a file and calls
write(2)for every transaction.
- The CUDA user library asks Linux to
mmap(2)the relevant control registers (DMA registers, PIO registers and some other MMIO registers) into the user space, and then operates the the GPU however it wants in user state.
- Something else?
I eliminated assumption #1 by running simple CUDA programs that copy data back and forth for 1000 times (and launches some empty kernels in between), where
strace(1) did not observe any calls to
Assumption #2 seem possible, since I observed with
time(1) that the amount of data transfered seem to scale with
user time instead of
sys time. So the program seem to be copying data in user state.
But it seems a bit weird. How could a user program be allowed to manipulate such important I/O control registers by itself?
I would be grateful for some professional ideas on this topic.