Tensorflow does not generate GPU tracing information

Question

I started a new machine learning project.

In according to this document (https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras)

TF with Tensorboard appears to support GPU profiling. So, i used the same code in my Jupyter Notebook for testing.

The sample code generates profiling resulting. However, there is no GPU tracing information in resulting file. (only CPU)

This is my main problem.

I am using two RTX 2080 TI graphic cards.

And also, they were working when running the code.

The sample code does not use MirroredStrategy. So, i could see the one of them was running.

At first, i thought Tensorboard was the problem. But,i realized soon that TF does not generate the GPU tracing information.

The image above is the resulting file (local.trace). There was no GPU data.

It is my system specification.

• OS ubuntu 18.04
• jupyter-client 5.3.4
• jupyter-core 4.6.1
• jupyter-tensorboard 0.1.10
• tensorflow-gpu 2.0.0
• tensorflow-estimator 2.0.1
• tensorflow-metadata 0.15.1
• tensorboard 2.0.2
• nVidia 410.104
• CUDA 10.0
• anaconda 4.7.12 (with python 3.6)

It looks irrelevant, but there was a warning message like the image below.

I have tested this on other PC and got the same resulting. It could be the GPU profiling is only supporting on Google Colab. (I am still confusing) Recently, I have searched it on google to fix the problem. I could not get still the answer.

Is there someone who is using GPU profiling on your own System instead of Google Colab?

Please give me piece of advices.

Answer 1

I figured out what caused the problem.

It was related with CUPTI(CUDA Profiling Tools Interface)

In contrast to Jupyter Notebook, there was a warning message when the code is running on Ubunto shell.

CUPTI error: CUPTI could not be loaded or symbol could not be found.

TF could not find CUPTI libraries. This is the main reason of the problem.

After adding the path to LD_LABRARY_PATH as below link, the problem is fixed!

https://stackoverflow.com/a/58752904/5553618