CODEWITHSUNDEEP
pythontensorflowgputensorflow2.0
raceee
raceee

Reputation: 517

Seperate gpu tensorlfow scripts running on seperate gpu's

I have recently purchased a second gpu with the thought that I could take advantage of tensorflow-gpu to run some tensorflow script on one gpu and then continue to run other gpu intensive programs on the other.

I was successful in running the tensorflow script on the gpu alone, but when I run another tensorflow script on another gpu I recieve this:

tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(15, 1024), b.shape=(1024, 1024), m=15, n=1024, k=1024 [Op:MatMul]

And when I run another graphicly intense program this throws:

2020-07-10 13:41:42.010777: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure
2020-07-10 13:41:42.021608: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:273] Unexpected Event status: 1

This error What is going on? Is my fantasy purely fictitious?

EDIT: It seems that when a gpu intensive program is currently running (i.e. another tensorflow script or a video game or something) the first error occurs. However, if the gpu script is up and running and then a video game or other gpu intensive program lauches, the script slows down and evenutally throws the second error.

Upvotes: 1

Views: 64

Answers (1)

theastronomist
theastronomist

Reputation: 1056

os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0" #select 0 for first GPU or 1 for second

Put this script at the very top of any python file to manually assign a GPU.

Upvotes: 1

Related Questions