6

I use the CUDA toolkit to perform some computations on my Nvidia GPUs. How to kill all processes that use a given GPU? (killing at once, i.e. without having to manually type the PIDs behind kill -9.)

E.g. killing all processes using GPU 2:

enter image description here

4

Following the Unix philosophy, you have a tool that lists processes using a given GPU, and a tool that kills processes. Combine them using shell constructs and text processing tools.

For example, to kill all the processes using GPU 2, you can execute the following command:

kill $(nvidia-smi | awk '$2=="Processes:" {p=1} p && $2 == 2 && $3 > 0 {print $3}')

or

kill $(nvidia-smi -g 2 | awk '$2=="Processes:" {p=1} p && $3 > 0 {print $3}')
  • $ kill $(nvidia-smi -g 1 | awk '$2=="Processes:" {p=1} p && $3 > 0 {print $3}') bash: kill: GPU: arguments must be process or job IDs bash: kill: PID: arguments must be process or job IDs – Mona Jalal Apr 9 '18 at 1:01
  • 1
    It did not worked for me, I guess there is syntax issue. – Rana Pratap Singh Nov 26 '18 at 12:13
  • the commands don't work – Curious Mar 8 at 19:03
0

maybe this is what you need:

kill -9 $(nvidia-smi | awk '$2 == "GPU" && $3 == "PID" {flag = 1} flag && $3 > 0 {print $2, $3}' | awk '$1 == 2 {print $2}')

For more complex conditions, you can change the condition statements of 2nd awk command. For example, the following command can be used to kill all processes that use GPU-0 to GPU-3 and PID > 1000:

kill -9 $(nvidia-smi | awk '$2 == "GPU" && $3 == "PID" {flag = 1} flag && $3 > 0 {print $2, $3}' | awk '$1 < 4 && $2 > 1000 {print $2}')

As you can see, kill -9 PIDs needs PIDs to shut processes down, and awk is used twice to find valid PIDs to kill.

More specifically, The 1st awk command will choose lines after the "GPU PID Type Process Name" line, and then print out lines of GPU ids and PIDs, with a space between each GPU id and PID. The 2nd awk will find specific GPU ids or PIDs, which in this case is to find all processes using GPU-2, and then print out the PID. Finally, kill -9 PIDs will kill the processes according to these PIDs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.