CODEWITHSUNDEEP
apache-sparkhdfstensorboard
Santosh Kumar
Santosh Kumar

Reputation: 791

Tensorboard: How to connect to hdfs directory

I have used TensorflowOnSpark to train a RNN model with tensorboard enabled(store_true). Summary events have been logged in a HDFS directory.

How can I visualize the RNN events(from hdfs directory) using tensorboard?

I tried to start tensorboard using hdfs log directory but it failed to start with message: "hdfs not supported".

Please let me know if anybody has any idea about it.

Upvotes: 0

Views: 1121

Answers (1)

mbeh
mbeh

Reputation: 11

If you're running Spark on cluster mode and not locally, you won't be able to view the events by running Tensorboard on the machine that launched the Spark job. This is because the summaries will be written to the HDFS directory of the Spark worker's machine.

Besides, if you had successfully enabled Tensorboard, the Spark cluster would have launched Tensorboard on your behalf, and printed the Tensorboard URL to your Spark logs.

To visualize events with tensorboard (using TensorFlowonSpark) :

  1. Launch a spark job with the --tensorboard flag or as defined in your spark.py file.

  2. Ensure you called TFCluster.run() with the argument tensorboard=store_true. If done correctly, this function will call TFSparkNode.run() that launches Tensorboard, and print out the Tensorboard URL here.

  3. Simply access the URL printed to your Spark logs.

Note that once the job succeeds or is killed, the machines terminate and your event files will be gone. In this situation, you might want to copy the event files to your local machine if you'd still like to view the events on Tensorboard after the job completes.

Upvotes: 1

Related Questions