CODEWITHSUNDEEP
python-3.xtensorflowmachine-learningtensorboard
G4bri3l
G4bri3l

Reputation: 5146

Why would Tensorflow not save run_metadata?

I was simply trying to generate a summary that would show the run_metadata as follows:

run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()
summary = sess.run([x, y], options=run_options, run_metadata=run_metadata)

train_writer.add_run_metadata(paths.logs, 'step%d' % step)
train_writer.add_summary(paths.logs, step)

I made sure the path to the logs folder exists, this is confirmed by the fact the the summary file is generated but no metadata is presetn. Now I am not sure a file is actually generated to be honest (for the metadata), but when I open tensorboard, the graph looks fine and the session runs dropdown menu is populated. Now when I select any of the runs it shows a progress bar "Parsing metadata.pbtxt" that stops and hangs right half way through.

This prevents me from gathering any more additional info about my graph. Am I missing something ? A similar issue happened when trying to run this tutorial locally (MNIST summary tutorial). I feel like I am missing something simple. Does anyone have an idea about what could cause this issue ? Why would my tensorboard hang when trying to load a session run data ?

Upvotes: 2

Views: 441

Answers (1)

G4bri3l
G4bri3l

Reputation: 5146

I can't believe I made it work right after posting the question but here it goes. I noticed that this line:

run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)

was giving me an error so I removed the params and turned it into

run_options = tf.RunOptions()

without realizing that this is what caused the metadata not to be parsed. Once I researched the error message:

Couldn't open CUDA library cupti64_90.dll

I looked into this Github Thread and moved the file into the bin folder. After that I ran again my code with the trace_level param, had no errors and the metadata was successfully parsed.

Upvotes: 1

Related Questions