CODEWITHSUNDEEP
apache-sparkhadoop-yarn
no123ff
no123ff

Reputation: 317

How to get status of Spark jobs on YARN using REST API?

A spark application can run many jobs. My spark is running on yarn. Version 2.2.0.

How to get job running status and other info for a given application id, possibly using REST API?

job like follows:
enter image description here

Upvotes: 0

Views: 2849

Answers (3)

Manju N
Manju N

Reputation: 1222

  • Based on app-name, just incase if you need

    curl http://YARN_UI_IP:8088/ws/v1/cluster/apps | jq '.apps | select(.app != null) | .app | map(select(.name == "my-spark-job-name"))' enter image description here

Upvotes: 0

OnkarK
OnkarK

Reputation: 41

This might be late but putting it for convenience. Hope it helps. You can use below Rest API command to get the status of any jobs running on YARN.

curl --negotiate -s -u : -X GET 'http://resourcemanagerhost:8088/ws/v1/cluster/apps/application_121766109986_12343/state'

O/P - {"state":"RUNNING"}

Throughout the job cycle the state will vary from NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED

You can use jq for a formatted output.

curl --negotiate -s -u : -X GET 'http://resourcemanagerhost:8088/ws/v1/cluster/apps/application_121766109986_12343'| jq .app.state

O/P - "RUNNING"

Upvotes: 1

tk421
tk421

Reputation: 5967

YARN has a Cluster Applications API. This shows the state along with other information. To use it:

$ curl 'RMURL/ws/v1/cluster/apps/APP_ID'

with your application id as APP_ID.

It provides:

enter image description here

Upvotes: 0

Related Questions