CODEWITHSUNDEEP
pythongpunvidiakeywordpid
salehinejad
salehinejad

Reputation: 7378

How to kill process on GPUs with PID in nvidia-smi using keyword?

How to kill running processes on GPUs for a specific program (e.g. python) in terminal? For example two processes are running with python in the top picture and kill them to see the bottom picture in nvidia-smi

For example two processes are running with python in the top picture and kill them to see the bottom picture in nvidia-smi

Upvotes: 49

Views: 146308

Answers (6)

salehinejad
salehinejad

Reputation: 7378

You can grep "python" in the nvidia-smi and then pass the PID to the kill -9 command, e.g.

sudo kill -9 $( nvidia-smi | grep 'python' | sed -n 's/|\s*[0-9]*\s*\([0-9]*\)\s*.*/\1/p' |  sed '/^$/d')

Upvotes: 12

Bitswazsky
Bitswazsky

Reputation: 4698

Keeping here as a reference. In case one wants to kill all running processes on the gpu, following command would work:

nvidia-smi --query-compute-apps=pid --format=csv,noheader | xargs -n1 kill -9

Upvotes: 3

Sadra
Sadra

Reputation: 2822

As one of other answers suggest you can use: (replace 5 with the column number where process id exists)

nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9

If you might have to use this a lot you can create an alias for the command: to do that do this you should edit ~/.bash_aliases file:

nano ~/.bash_aliases

and add the following line to it and save the file:

alias killgpuprocess="nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9"

then (just needed this time):

source ~/.bashrc

Then if you run

killgpuprocess

it will kill the existing processes on GPU(s).

Upvotes: 8

Soma Siddhartha
Soma Siddhartha

Reputation: 349

I guess the question is already answered when nvidia-smi shows processes occupying GPU mem. For me, even though nvidia-smi wasnt showing any processes, GPU memory was being used and I wanted to kill them.

The way to go in this case was to use the fuser command to find out the processes using the particular GPU device. In my case I wanted to kill all the processes using the GPU device 3. This can be done using the command :

sudo fuser -k /dev/nvidia3

You can use -ki to kill the processes interactively.

Upvotes: 12

Snehal Rajput
Snehal Rajput

Reputation: 436

Use nvidia-smi or top command to see processes running and to kill command:

sudo kill -9 PID

Upvotes: 19

Ainz Titor
Ainz Titor

Reputation: 1557

The accepted answer doesn't work for me, probably because nvidia-smi has different formats across different versions/hardware.

I'm using a much cleaner command:

nvidia-smi | grep 'python' | awk '{ print $3 }' | xargs -n1 kill -9

You can replace $3 in the awk expression to fit your nvidia-smi output. It is the n-th column in which the PIDs occur.

Upvotes: 68

Related Questions