CODEWITHSUNDEEP
loggingtensorflowtensorboard
opus111
opus111

Reputation: 2854

No Tensorboard logging in Tensorflow Estimator framework

I am using Estimator to train my net. I want to monitor with Tensorboard. Blogs for Estimator claim things like:

"The training will output information like the global step, loss, 
and accuracy over time on the terminal output. Besides this, the
Experiment and Estimator framework will log certain statistics to 
be visualized by TensorBoard."

https://medium.com/onfido-tech/higher-level-apis-in-tensorflow-67bfb602e6c0

My process does indeed create and event file, but there is nothing in it.

These tags are in checkpoints/1504359209.469093:
audio -
histograms -
images -
scalars -
tensor -

What controls which scalars etc. Estimator writes, and how often it writes them?

Upvotes: 0

Views: 991

Answers (2)

opus111
opus111

Reputation: 2854

Thank you. That's very helpful.

I have, already got it working using the default scalars, but your suggestions will allow me to add new ones.

In my case, the problem was in the data input. I copied existing code which uses a DataSet. Here is my code 

        # Build dataset iterator
        dataset = tf.contrib.data.Dataset.from_tensor_slices(
        # dataset = tf.contrib.data.Dataset.from_tensors(
            (fingerprints_placeholder, labels_placeholder))
        #dataset = dataset.repeat(None)  # Infinite iterations
        #dataset = dataset.shuffle(buffer_size=10000)
        dataset = dataset.batch(batch_size)
        iterator = dataset.make_initializable_iterator()

Notice the commented out lines. For some reason they cause the code to run forever with no output. Removing them allowed everything to work as advertised.

Upvotes: 0

user1536634
user1536634

Reputation: 36

Firstly, the documentation is not very clear on this point. To avoid writing a tutorial, here are a few points that you might find helpful, although I should say I am not quite clear on how your code is currently set up.

  • You are not looking to create a FileWriter. Rather, you're just going to add tf.summary.scalar and tf.name_scope annotations to your model_fn.
  • You are not looking to pass anything back from sess.run. That was my first guess, and I can confirm you can keep your Estimator and don't need to restructure.
  • You should be looking at the line that creates your estimator (estimator.Estimator(model_fn=...)). On this line you are specifying your model_dir, which is where you will point tensorboard's logdir parameter. Estimators automatically log to the model_dir, they don't need a FileWriter. You probably already know this since you can see the event output, but I mention it in case it uncovers some inconsistency between what we are doing.
  • You should also look at the RunConfig you pass to your estimator. My contains the key save_summary_steps, and I think that controls how often writes occur.

Upvotes: 1

Related Questions