How to flush GPU memory using CUDA on WSL2

Question

I have interrupted the training of the model in PyTorch on CUDA, which I've run on Windows Subsystem for Linux 2 (WSL2). The dedicated GPU memory of NVIDIA GeForce RTX 3080Ti was not flushed.

What I have tried:

• gc.collect() and torch.cuda.empty_cache() does not resolve the problem (reference)

• When running numba.cuda.select_device(0) to potentially cuda.close(), the notebook hangs (reference)

• After running nvidia-smi to potentially reset the GPU (reference), the command prompt hangs

• Win + Ctrl + Shift + B to reset the graphics stack in Windows does not help (reference)

• Restarting the notebook kernel as well as restarting the notebook server does not help

• Physical reset is not available

UPDATE:

Running nvidia-smi in the command prompt on Windows (not on WSL2) yields the following

Answer 1

I don't know your actual environment.

suppose that you use anaconda window-venv

1. On cmd >nvidia-smi shows following.

2. Check pid of python process name(>envs\psychopy\python.exe).
3. On cmd taskkill /f /PID xxxx

this could be help.

and you don't want doing like this.

if you feeling annoying you can run the script on prompt, it would be automatically flushing gpu memory.