CODEWITHSUNDEEP
apache-spark
poiuytrez
poiuytrez

Reputation: 22546

How to keep the Spark web UI alive?

After the end of the execution of Spark submit, the Spark web UI is killed. Is there a way to keep it alive?

I am using Spark 1.2.1.

Upvotes: 26

Views: 18580

Answers (8)

Ankit
Ankit

Reputation: 4644

As Justin Pihony mentioned above, "The web UI is intrinsically tied to the SparkContext". It is alive as long as SparkContext is alive.

What we can try is to delay the calling of stop() on SparkContext object. I've tried this and it works well for me:

//Perform your spark transformations and actions
  ..............................
//In the end
ScheduledExecutorService service = Executors.newScheduledThreadPool(1);
service.schedule(() -> sparkSession.stop(), 5, TimeUnit.MINUTES);
service.shutdown();

You can change the time according your need.

Upvotes: 0

Praveen Kumar K S
Praveen Kumar K S

Reputation: 3074

Solution to your Answer

  1. Before SparkContext to stop, use Thread.sleep(86400000). This will keep 24 hours active of your Spark UI until you kill the process. This is not an advisable solution.

    SparkConf conf = new SparkConf().setAppName("Simple Application").setMaster("local[*]");
JavaSparkContext sc = new JavaSparkContext(conf);
Thread.sleep(86400000); //throws InterruptedException. 
sc.stop();

    1. Best Solution is to log Spark events somewhere and use spark history server.

      spark.eventLog.enabled=true
$ ./sbin/start-history-server.sh

Upvotes: 0

Régis B.
Régis B.

Reputation: 10618

While locally testing Spark applications written in Python, I add this as a little hack to the end of my apps:

raw_input("Press ctrl+c to exit")

When running on a YARN cluster manager, I use the history manager available on port 18080.

Upvotes: 0

Hari
Hari

Reputation: 681

If you are testing in local mode i.e. using IDEA or Eclipse, one way is to do is as following. 

System.in.read();
spark.stop(); // spark --> SparkSession

This will ensure that the UI is accessible as long as you want. Just hit enter on the IDEA/Eclipse console to terminate the application

Upvotes: 13

oaksharks
oaksharks

Reputation: 1

may be you can add the line :

new Scanner(System.in).nextLine()

make sure it run in driver

Upvotes: 0

bluenote10
bluenote10

Reputation: 26729

To add a newbie friendly step-by-step solution of how to work with the history server:

  • In the spark distribution folder try to start the history server by:

    ./sbin/start-history-server.sh

    By default the history server will try to monitor /tmp/spark-events for logs and unfortunately it will crash if the path doesn't exist. So if you get an error you may have to mkdir /tmp/spark-events first. You can check the history server logs in ./logs to see details in case of trouble.

  • In order for a context to persists its event log, you have to enable event logging. This can be done either programmatically or by editing ./conf/spark-defaults.conf (copy the template if it doesn't exist yet), and uncommenting/adding the line:

    spark.eventLog.enabled true

    Running a spark-submit should result in event log folders like /tmp/spark-events/local-1465133722470.

  • Access the history server UI, typically from http://localhost:18080

Upvotes: 18

Justin Pihony
Justin Pihony

Reputation: 67135

The web UI is intrinsically tied to the SparkContext, so if you do not call .stop and keep your application alive, then the UI should remain alive. If you need to view the logs, then those should still be persisted to the server, though. It might make for an interesting feature to keep the web server portion open for a time period, or some other view, though, possibly a feature request?

From SparkContext.scala

// Initialize the Spark UI
private[spark] val ui: Option[SparkUI] =
if (conf.getBoolean("spark.ui.enabled", true)) {
  Some(SparkUI.createLiveUI(this, conf, listenerBus, jobProgressListener,
    env.securityManager,appName))
} else {
  // For tests, do not enable the UI
  None
}


/** Shut down the SparkContext. */
  def stop() {
    SparkContext.SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
      postApplicationEnd()
      ui.foreach(_.stop())
      ...
    }
  }

UPDATE - BETTER ANSWER

I had forgotten about the spark history server. That is something you might want to look into

Upvotes: 3

Josh Rosen
Josh Rosen

Reputation: 13841

You can use Spark's event logging and history server features to view the UIs of completed applications; see https://spark.apache.org/docs/latest/monitoring.html for more details.

Upvotes: 6

Related Questions