Set up TensorBoard to run eval.py job for TensorFlow object detection models in Google Colab

Question

I am new to Deep Learning field. I would like to use TensorBoard while I run an object detection model in Google Colab.

I have referred couple of discussion threads(How to run eval.py job for tensorflow object detection models)

But did not get answer to my specific query.

I am giving below the steps I followed:

Process 1:

1.Downloaded ngrok and unzipped

!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
!unzip -o ngrok-stable-linux-amd64.zip

2.Specified log dir and fired tensorboard in the background:

LOG_DIR = '/content/models/research/training'
get_ipython().system_raw(
    'tensorboard --logdir {} --host 0.0.0.0 --port 6006 &'
    .format(LOG_DIR)
)

3.Ran ngrok to tunnel TensorBoard through port 6006:

get_ipython().system_raw('./ngrok http 6006 &')

4. Generated TensorBoard link:

   ! curl -s http://localhost:4040/api/tunnels | python3 -c \ "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"

Issue:

i. When I opened the link, a blank page opened with notification:

*No dashboards are active for the current data set.
Probable causes:
You haven’t written any data to your event files.
TensorBoard can’t find your event files.*

ii. While the training was in progress, I went back and refreshed tensorboard and it looked blank

Also, at any given instance my log directory(which is training directory) had following files: where is the log file?

My question:

How I will get to see training progress in the TensorBoard? Is there anything wrong with the approach I followed?

Process 2:

As @cookiemonster suggested:

I tried the following set up to open the TensorBoard from the Colab cell itself:

!pip install tb-nightly
%load_ext tensorboard

%tensorboard --logdir '/content/models/research/training'

Output:

Just an orange bar came

How can I see training status in this process while the training is on.

Final question:

My main objective is to check different evaluation matrices by changing the value of 'matrics_set' in the config file and then by running the eval.py I need to check the results.

I read in one thread that the 'The results are output to an events summary file prefixed with events.out.tfevents, which you can visualize using TensorBoard'

(reference: Run object detection evaluation protocols (tensorflow))

My question:

If I just run the following code will log file get generate in the log directory I specified?

!python eval.py --logtostderr --pipeline_config_path=<path to training directory>/faster_rcnn_inception_v2_pets.config --checkpoint_dir=<path to training directory> --eval_dir=<path to log directory>

With the current basic issue I am facing in viewing the tensorboard, how can I achieve my main objective of viewing the result of running eval.py?

Answer 1

FYI the new tensorboard could be run directly in the notebook, so you can run

!pip install tb-nightly
%load_ext tensorboard
%tensorboard --logdir "/content/runs"

or you mount google drive and can run it in another colab notebook (maybe using only CPU) to avoid messing up the installed libraries

%tensorboard --logdir "/content/drive/My Drive/Colab Runs"