CODEWITHSUNDEEP
apache-sparkhadoop-yarnhortonworks-data-platformapache-zeppelin
vero
vero

Reputation: 1015

Zeppelin persists job in YARN

When I run a Spark job from Zeppelin, the job finishes with success, but it stays in YARN on mode running. The problem is the job is taking a resource in YARN. I think that Zeppelin persists the job in YARN.

How can I resolve this problem?

Thank you

Upvotes: 3

Views: 770

Answers (3)

cenote
cenote

Reputation: 251

Just close your spark context so that the spark job will get the status FINISHED.

Your memory should be released.

Upvotes: 0

rag_sri
rag_sri

Reputation: 38

The easiest and straight-forward solution is to restart the spark interpreter. But as Rick mentioned if you should use the spark dynamic allocation, an additional step of enabling spark shuffle service on all agent nodes is required(this by default is disabled).

Upvotes: 1

Rick Moritz
Rick Moritz

Reputation: 1518

There are two solutions.

The quick one is to use the "restart interpreter" functionality, which is misnamed, since it merely stops the interpreter. In this case the Spark job in Yarn.

The elegant one is to configure Zeppelin to use dynamic allocation with Spark. In that case the Yarn application master will continue running, and with it the Spark driver, but all executors (which are the real resource hog) can be freed by Yarn, when they're not in use.

Upvotes: 4

Related Questions