CODEWITHSUNDEEP
pythontensorflowkerasdeep-learningtensorboard
samuraikmc
samuraikmc

Reputation: 33

Tensorboard Debugger V2 not displaying data

I am trying to debug a Nan training issue, and thought to use Tensorboard Debugger V2 to do so. I implemented it with this line at the top of my training code:

tf.debugging.experimental.enable_dump_debug_info(OUTPUT_FOLDER + 'tensorboard' +
                                                    "/debug/tfdbg2_logdir",
                                                    tensor_debug_mode="FULL_HEALTH",
                                                    circular_buffer_size=-1)

However, no data shows on Tensorboard even though the folder is being populated with log data. Both are version 2.9.0 enter image description here

Upvotes: 2

Views: 193

Answers (1)

Breck Emert
Breck Emert

Reputation: 36

My issue when encountering this problem was that my log files were 0 KB; check to make sure your logs are actually valid. Mine worked when using NO_TENSOR mode (the default), and I still have to troubleshoot why the other methods aren't working.

Also, please check if fully disabling eager mode works:

model.compile(run_eagerly=True)
# also comment out any @tf.function

Upvotes: 0

Related Questions