CODEWITHSUNDEEP
tensorflowprofilingtensorboard
Eduardo Reis
Eduardo Reis

Reputation: 1971

Tensorboard example not reproducing GPU profile

When running this notebook on Colab:

https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras

I am not getting the same profile output for profiling: on my side no GPU device is shown.

This is what is expected: enter image description here

This is what I am getting: enter image description here

Upvotes: 4

Views: 1432

Answers (2)

danodonovan
danodonovan

Reputation: 20373

I was seeing the same problems (no GPU profiling trace, running tensorflow r2.1.0) It turns out that my tensorflow library hadn't been compiled with CUPTI (despite having CUPTI installed locally).

Errors like

2020-01-29 14:09:01.088485: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1329] function cupti_interface_->EnableCallback( 0 , subscriber_, CUPTI_CB_DOMAIN_DRIVER_API, cbid)failed with error CUPTI could not be loaded or symbol could not be found.
2020-01-29 14:09:01.088566: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:88]  GpuTracer has collected 0 callback api events and 0 activity events.

were the smoking gun. To fix this (for my required version of tensorflow) I had to update the CUPTI library paths - (ie add)

$ echo '/usr/local/cuda/extras/CUPTI/lib64' >> /etc/ld.so.conf.d/cupti.conf

(or wherever your CUPTI libs are installed), reload your config

$ sudo ldconfig -v

and then re ./configure and rebuild tensorflow from source.

I (believe) the problem lies with the TF binary being compiled with CUPTI, and this could be because NVIDIA apt install CUPTI and don't set it up correctly - however I would love to be corrected!

Upvotes: 2

Nicholas Kalscheuer
Nicholas Kalscheuer

Reputation: 663

This may be an issue with GPUs not being available in the Colab environment. So you may have to check back later.

Referenced from this question: Google-colaboratory: No backend with GPU available

Upvotes: 0

Related Questions