CODEWITHSUNDEEP
apache-sparkhadoopgoogle-cloud-platform
Franco Victorio
Franco Victorio

Reputation: 217

Spark jobs not showing up in Hadoop UI in Google Cloud

I created a cluster in Google Cloud and submitted a spark job. Then I connected to the UI following these instructions: I created an ssh tunnel and used it to open the Hadoop web interface. But the job is not showing up.

Some extra information:

  1. If I connect to the master node of the cluster via ssh and run spark-shell, this "job" does show up in the hadoop web interface.
  2. I'm pretty sure I did this before and I could see my jobs (both running and already finished). I don't know what happened in between for them to stop appearing.

Upvotes: 1

Views: 170

Answers (1)

Franco Victorio
Franco Victorio

Reputation: 217

The problem was that I was running my jobs in local mode. My code had a .master("local[*]") that was causing this. After removing it, the jobs showed up in the Hadoop UI as before.

Upvotes: 1

Related Questions