CODEWITHSUNDEEP
amazon-web-servicesapache-sparkapache-spark-sqlemramazon-emr
Kristian
Kristian

Reputation: 21840

Running an EMR Spark script, and the Spark UI SQL tab disappears

When running spark sql jobs on local mode, my Spark UI has a SQL tab.

enter image description here

When running that same job on AWS EMR, the Spark UI's SQL job is no longer there?

enter image description here

is there a code reason why this would not be there in Amazon's EMR version of the Spark UI?

Upvotes: 0

Views: 493

Answers (1)

Kristian
Kristian

Reputation: 21840

There are two different UI's for Spark in EMR: 1. the Spark UI, and 2. the Spark History Server.

They look the same, but they are served from different URLs, and one of them continues to be served even after an application completes, as long as the EMR cluster is still up and running.

The history server AFAIK does not have a SQL tab

Upvotes: 1

Related Questions