CODEWITHSUNDEEP
tensorflowtensorboard
P. Camilleri
P. Camilleri

Reputation: 13218

All runs are not visible on TensorBoard

I am using tensorboard to visualize three runs. I have a folder, logs, which contains three files: 

2016-03-18_22-11-12
2016-03-18_22-11-27
2016-03-18_22-23-46

when I run tensorboard --logdir . (from logs), only 2016-03-18_22-23-46 is visible:blu

And if I delete 2016-03-18_22-23-46 from logs and restart tensorboard, then only 2016-03-18_22-11-27 is visible. Any idea of what's happening here?

Edit: the log files are (to my surprise), quite big: here is the result of du -h:

1,1G    ./2016-03-18_22-23-46
925M    ./2016-03-18_22-11-12
934M    ./2016-03-18_22-11-27
2,9G    .

EDIT: The above run structure can be obtained via logging and checkpointing into subdirectories of the tensorboard log-dir:

run_time = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
FLAGS.train_dir = '/datalab/tf_runs/' + run_time #Directory to put the training data.
summary_writer = tf.train.SummaryWriter(FLAGS.train_dir, sess.graph_def)

Upvotes: 12

Views: 10707

Answers (3)

MJimitater
MJimitater

Reputation: 939

TL;DR: Close any currently running jupyter notebook / python file that has Tensorboard callbacks.

I had the same problem, only one run was visualized in Tensorboard and multiple weren't listed. I waited and refreshed some time, as @etarion pointed out in his answer, but that didn't make the other runs appear.

Then I found out what was happening: I had just finished running training in my notebook file. Then I wanted to open all previous runs in Tensorboard, and the only one appearing was the current one I had just trained. I first had to close the notebook file and then running Tensorboard showed all runs!

For some reason the notebook still being active led Tensorboard only take this run.

Upvotes: 1

Vijay Gunasekaran
Vijay Gunasekaran

Reputation: 161

Another solution is to use the --max_reload_threads option as follows:

tensorboard --logdir=runs --max_reload_threads 4

The no. of threads could be determined based on the fact that one thread can parse one run at a time.

Upvotes: 6

etarion
etarion

Reputation: 17159

Tensorboard takes a while to parse the log files. If you refresh the graph, you can see it showing more and more iterations. When it finished parsing one, the next run will pop up.

It is a bit annoying that there is no visual indication for this, maybe this is worth a feature request at https://github.com/tensorflow.

Upvotes: 18

Related Questions