CODEWITHSUNDEEP
pythontensorflowhistogramtensorboardsummary
hjsg1010
hjsg1010

Reputation: 173

How to draw weights histogram on tensorboard?

I have a train.py and a model.py for 3D classification deep learning model. I define the model in model.py. In train.py, I import model from model.py and start training.

I visualized training loss or accuracy, by using tf.summary.scalar function in train.py.

However, I'm having trouble when I am trying to plot weight's histogram. I find tf.summary.histogram function, but I can't figure out where to put this function (in train.py or model.py?).

In model.py, I defined layers using tf.contrib.layers function. (sorry for my poor English skills)

PS:
I saved my model as .ckpt format. Can I plot weight histogram by loading this saved model?

Upvotes: 2

Views: 3186

Answers (1)

End-2-End
End-2-End

Reputation: 921

I'll try to answer each of your questions separately.

I'm having trouble when I am trying to draw weight's histogram.

Ideally, you define weights and biases using tf.get_variable and add them to summary histograms.

For example: 

with tf.name_scope("layer1"):
    W1 = tf.get_variable("W1", shape=[input_size, hidden_layer_neurons],
                         initializer=tf.contrib.layers.xavier_initializer())
    layer1 = tf.matmul(X, W1)
    layer1_act = tf.nn.tanh(layer1)
    tf.summary.histogram("weights", W1)

If needed, we can also add histograms of layer-outputs and activation-outputs: 

    tf.summary.histogram("layer-outputs", layer1)
    tf.summary.histogram("activation-outputs", layer1_act)

But since you're using tf.contrib.layers, you don't have such a provision as contrib.layers takes care of creating weights and biases for you. In such a case, you can have a look at tf.trainable_variables(); This should contain all the trainable variables from your graph, that contains all the weights and biases of the network.

You can define a simple function like this:

def add_hist(train_vars):
    for i in train_vars:
        name = i.name.split(":")[0]
        value = i.value()
        tf.summary.histogram(name, value)

I can't figure out where to put this function (in train.py? or model.py?)

Scalar summaries like loss and accuracy are obtained during training, hence those are included in train.py; Whereas weights and biases are a part of your core-model, and are hence to be included in model.py

So in your model.py, include this before you use tf.summary.merge_all()

train_vars = tf.trainable_variables()
add_hist(train_vars)

Can I draw weight histogram by load this saved model?

These histograms generally show the distribution of entities (weights or activations etc) during the training. These plots are often mainly used for insights on how these distributions are changing over time, if or not the parameters or their gradients are saturated, any clear steps to be taken to improve etc.

Since your saved checkpoint (.ckpt) is only supposed to contain the final weights and biases, only the final distribution is obtained and not the histogram plot over the entire training period.

Hope this helps.

Upvotes: 3

Related Questions