CODEWITHSUNDEEP
pythontensorflowkerastensorboard
Atsushi Sakai
Atsushi Sakai

Reputation: 309

How to show TensorBoard's CPU/memory usage (RunMetadata) for Keras

I want to view the CPU/memory usage in TensorBoard with Keras. For this purpose, I need to execute the method of add_run_metadata. But I cannot found the way to pass the add_run_metadata method in Keras's TensorBoard callback. Is this good way to implement CPU/memory usage for Keras.

Reference

EDIT: I encountered the same problem. I'm editing to share how I attempted to approch this.

I changed the keras source for: callbacks.py, and replaced this line in on_epoch_end() with -

run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()
result = self.sess.run([self.merged], feed_dict=feed_dict, options=run_options, run_metadata=run_metadata)
self.writer.add_run_metadata(run_metadata, 'epoch%d_step%d' % (epoch, i))

However I end up with the following error:

...\tensorflow\stream_executor\dso_loader.cc:141] Couldn't open CUDA library cupti64_90.dll
...\tensorflow/stream_executor/lib/statusor.h:212] Non-OK-status: status_ status: Failed precondition: could not dlopen DSO: cupti64_90.dll; dlerror: cupti64_90.dll not found

Which is puzzling to me as it seems to be related to the proper installation of cuda and not related in any obvious (to me) way to the change I made.

I'm using keras version 2.1.6 and tensorflow version 1.6.0

Upvotes: 4

Views: 2983

Answers (1)

Mark Loyman
Mark Loyman

Reputation: 2180

The solution is to run the Keras model in a TF session, and is based on the blog post: keras-as-a-simplified-interface-to-tensorflow#calling-keras-layers-on-tensorflow-tensors. Bellow is a detailed full and minimal working example.

First of all, dummy generation of data:

data.py

import numpy as np
def load_data(n = 1000):
    x = np.random.rand(n, 100)
    y = np.sum(x, axis=1, keepdims=True)
    return x, y

The core idea is to run the model in TF session, so the main code is pure TF, and only the model itself is defined with Keras. For this to work (following the above mentioned tutorial):

  1. The model needs to be built on top of a tf.placeholder, instead of the keras.layers.Input.
  2. Remain as a tensor, and not compiled into a keras.models.Model.

from keras.layers import Dense

model.py

def load_network(input_tensor):    
    x = Dense(100, activation='relu')(input_tensor)
    x = Dense(100, activation='relu')(x)
    x = Dense(1, activation='sigmoid')(x)    
    return x

And the TF session that runs the keras model (a clean, but full, version of the TensorBoard tutorial):

run_runtime_stats.py

import tensorflow as tf
sess = tf.Session()    
from keras import backend as K
from keras.objectives import mean_squared_error
K.set_session(sess)    
from model import load_network
from data import load_data

# load your keras model as a tf.Tensor
input = tf.placeholder(tf.float32, shape=(None, 100))  # is passed as input to our keras layers  
labels = tf.placeholder(tf.float32, shape=(None, 1))
net = load_network(input)  # type(net) == tf.Tensor

loss = tf.reduce_mean(mean_squared_error(labels, net))
opt = tf.train.GradientDescentOptimizer(0.1).minimize(loss)

writer = tf.summary.FileWriter(r'./logs', sess.graph)

sess.run(tf.global_variables_initializer())
with sess.as_default():    
    x, y = load_data(64)

    run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
    run_metadata = tf.RunMetadata()

    sess.run([opt], 
         feed_dict={input: x, labels: y},
         options=run_options,
         run_metadata=run_metadata)

    writer.add_run_metadata(run_metadata, 'runtime-statistics')
    writer.close()

Upvotes: 1

Related Questions