CODEWITHSUNDEEP
pythontensorflowmxnet
Ivan Kovtun
Ivan Kovtun

Reputation: 703

Wrong gpu order in mxnet and tensorflow

My desktop has 2 gpu installed: 1080 and 1080Ti nvidia-smi shows that gpu-0 is 1080 and gpu-1 is 1080Ti

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0 Off |                  N/A |
| 26%   57C    P2    53W / 215W |    696MiB /  8119MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:02:00.0 Off |                  N/A |
| 55%   70C    P2   204W / 250W |   8641MiB / 11178MiB |     28%      Default |
+-------------------------------+----------------------+----------------------+

Right now both tensorflow and mxnet use reversed order: 1080ti when I specify gpu=0 and 1080 when I specify gpu=1.

Why is it happening and how to synchronize tensorflow and mxnet gpu order with nvidia-smi gpu order?

Code snippets for mxnet:

mod = mx.mod.Module(symbol, label_names=None, context=mx.gpu(0))

For tensorflow I use environment variable

CUDA_VISIBLE_DEVICES="0"

Upvotes: 3

Views: 221

Answers (1)

stefan.schroedl
stefan.schroedl

Reputation: 866

Set

export CUDA_DEVICE_ORDER=PCI_BUS_ID.

Also see this question

Upvotes: 2

Related Questions