How would one go about identifying processes (gpu processes specifically) that haven't done anything in the last 2 hours - for example?
The problem I'm having stems from multiple users trying to use a single GPU resource. What happens is that a user will start an interactive python process that consumes 99% of GPU memory and then forgets about it or goes away for awhile (or perhaps the function he/she calls raises an exception and they don't notice), and the process continues to occupy that memory even though it's idle, which prevents other users from using the GPU resource. What I would like to do is have a cronjob that periodically runs to identify these kinds of idle processes, and then sends out an email warning or perhaps kills the process entirely. Is there some sort of timestamp buried within procfs that tells me when a process last did anything?