Tensorboard not giving all the variables output for Visualization Python

Question

Here are the variables that I have initialized for the tensorflow training, validation and testing.

index_in_epoch = 0;
perm_array  = np.arange(x_train.shape[0])
np.random.shuffle(perm_array)

# function to get the next batch
def get_next_batch(batch_size):
    global index_in_epoch, x_train, perm_array   
    start = index_in_epoch
    index_in_epoch += batch_size

    if index_in_epoch > x_train.shape[0]:
        np.random.shuffle(perm_array) # shuffle permutation array
        start = 0 # start next epoch
        index_in_epoch = batch_size

    end = index_in_epoch
    return x_train[perm_array[start:end]], y_train[perm_array[start:end]]

# parameters
n_steps = seq_len-1 
n_inputs = x_train.shape[2]#4 

n_neurons = 200
n_outputs = y_train.shape[1]#4
n_layers = 2
learning_rate = 0.001

batch_size = 50
n_epochs = 100#200 
train_set_size = x_train.shape[0]
test_set_size = x_test.shape[0]

tf.reset_default_graph()

X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
y = tf.placeholder(tf.float32, [None, n_outputs])

# use LSTM Cell with peephole connections
layers = [tf.contrib.rnn.LSTMCell(num_units=n_neurons, 
                                 activation=tf.nn.leaky_relu, use_peepholes = True)
         for layer in range(n_layers)]

multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)

stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons]) 
stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)
outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])
outputs = outputs[:,n_steps-1,:] # keep only last output of sequence

loss = tf.reduce_mean(tf.square(outputs - y)) # loss function = mean squared error 
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate) 
training_op = optimizer.minimize(loss)

This is the way I am training and validating the model and also collecting the values to get displayed via tensorboard:

saver = tf.train.Saver()
with tf.Session() as sess: 
    sess.run(tf.global_variables_initializer())

    for iteration in range(int(n_epochs*train_set_size/batch_size)):
        x_batch, y_batch = get_next_batch(batch_size) # fetch the next training batch 
        writer = tf.summary.FileWriter("outputLogs", sess.graph)
        sess.run(training_op, feed_dict={X: x_batch, y: y_batch}) 
        writer.close()
        if iteration % int(5*train_set_size/batch_size) == 0:
            mse_train = loss.eval(feed_dict={X: x_train, y: y_train}) 
            mse_valid = loss.eval(feed_dict={X: x_valid, y: y_valid}) 
            print('%.2f epochs: MSE train/valid = %.10f/%.10f'%(
                iteration*batch_size/train_set_size, mse_train, mse_valid))
            save_path = saver.save(sess, "models\\model"+str(iteration)+".ckpt")

But after running the command : tensorboard --logdir outputLogs I am only getting the graph and not all the other values graph, like the loss , error or other variables whatever I can display while training. See the below image:

Kindly, help me visualize all the variable parameter or inputs so that I can see the on tensorboard and can make the training workable for me.

Answer 1

You have to tell TensorFlow that you wanna keep track of your loss. You're only adding the graph to the writer. For example, you can do this to track your loss:

loss = ... (your def)
tf.summary.scalar('MyLoss', loss)

# ... maybe add some other variables (you can also make histograms, images, etc. via tf.summary.historam(...))

summ = tf.summary.merge_all()

In your session, you can create the writer as you did. Then you have to eval the summary operation and add it to the writer. You should create the writer outside the training loop, however, as you don't want a writer for every iteration. You provide the iteration as argument in the add_summary method.

saver = tf.train.Saver()
with tf.Session() as sess: 
    sess.run(tf.global_variables_initializer())
    writer = tf.summary.FileWriter("outputLogs", sess.graph)

    for iteration in range(int(n_epochs*train_set_size/batch_size)):

        x_batch, y_batch = get_next_batch(batch_size) # fetch the next training batch 

        [_, s] = sess.run([training_op, summ], feed_dict={X: x_batch, y: y_batch}) 

        writer.add_summary(s, iteration)

        if iteration % int(5*train_set_size/batch_size) == 0:
            mse_train = loss.eval(feed_dict={X: x_train, y: y_train}) 
            mse_valid = loss.eval(feed_dict={X: x_valid, y: y_valid}) 
            print('%.2f epochs: MSE train/valid = %.10f/%.10f'%(
                iteration*batch_size/train_set_size, mse_train, mse_valid))
            save_path = saver.save(sess, "models\\model"+str(iteration)+".ckpt")

Your training code should look like this.