CODEWITHSUNDEEP
pythonmachine-learningtensorflow
user1691278
user1691278

Reputation: 1885

Tensorflow - show the gradients on all the neurons (not just the input variables)

I am implementing a neural network using the Adam optimizer which utilizes gradients. Here is a brief look on my code

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

And this is how I feed in the data:

with tf.Session() as sess:
# Initialize variables
sess.run(init)

# Training cycle
for epoch in range(150):
    avg_cost = 0.
    total_batch = int(X_train.shape[0]/batch_size)
    batch_range = list(range(batch_size, int(X_train.shape[0]),batch_size))
    # Loop over all batches
    i = 0
    while i < total_batch - 1:
        start_idx = batch_range[i]
        end_idx = batch_range[i+1]
        batch_x, batch_y = X_train.iloc[start_idx:end_idx,:], y_train.iloc[start_idx:end_idx,:]
        # Run optimization op (backprop) and cost op (to get loss value)
        _, c = sess.run([optimizer, cost], feed_dict={x: batch_x,
                                                      y: batch_y})
        # Compute average loss
        avg_cost += c / total_batch
        i = i + 1

However, I would like to show gradients on all the neurons in the hidden layers (i.e. all the gradients used in the backprop). I tried to look up the relevant function, but all the functions that I found only compute the gradients with respect to the input variables. Can I please get a hint as to how to proceed?

Upvotes: 0

Views: 984

Answers (1)

Shanqing Cai
Shanqing Cai

Reputation: 3876

See the builtin debugger of TensorFlow called tfdbg (available in versions 0.12+). It exposes all the intermediate Tensors (along with graph structure) during Session.run() calls.

See a walk-through at: https://www.tensorflow.org/versions/master/how_tos/debugger/

The basic workflow is as follows. First you wrap your Session object right before the training Session.run() is called.

from tensorflow.python import debug as tf_debug
sess = tf_debug.LocalCLIDebugWrapperSession(sess)

Then when the sess.run() occurs, you will drop to the tfdbg> command line interface. Issue the tfdbg> run command and you will then see a list of all intermediate Tensors generated during the sess.run() call, including the gradients.

If you want to access the gradients programmatically, instead of interactively, tfdbg has a Python API available as well. See: https://www.tensorflow.org/versions/master/api_docs/python/tf_debug/

Upvotes: 2

Related Questions