CODEWITHSUNDEEP
hadoopoozieoozie-workflow
techie
techie

Reputation: 363

Get spark application id based on oozie job id

I am trying to get spark application id from unix based on oozie id. I am able to get map reduce job id when i try with oozie -info <oozie_id>@<action_name>. How can I get spark application id

Upvotes: 1

Views: 3225

Answers (2)

SnigJi
SnigJi

Reputation: 1410

There are two IDs for a oozie Spark action. And both aren’t same.

  1. Oozie launcher job ID.
  2. Spark job ID associated with the launcher.

To get both the ids:

  1. oozie -info <oozie_id>@<action_name> will give you oozie launcher ID.
  2. oozie job -info <oozie-launcher-id> | grep racking this will give you spark application id using oozie launcher ID that you will get from the 1st command.

Note: grep racking actually stands for grep tracking but at the moment I forgot if it starts with capital T or small t but racking will do the job.

Upvotes: 0

sathya
sathya

Reputation: 2072

From Web UI:

to get to a spark application log,

  1. go to oozie's web console and find the hadoop job Id of that action such as : job_202007171535_0223

    you can get app id by replacing job with application in job id, such as application_202007171535_0223 from job_202007171535_0223

  2. using the hadoop job/application id from step 1, go to Yarn Resource Manager WebUI to look into that logs(common for hive/hadoop/spark jobs which are managed by YARN)

  3. access to Spark UI for the debugging DAG,Stages,fine tuning using the app id, such as application_202007171535_0223

From command line interface, the above steps would be:

  1. Run oozie cmd to get hadoop job id:

    oozie job -info <your job id here>

  2. look for External Id in the output, such as : job_202007171535_0223

    you can get app id by replacing job with application in job id, such as application_202007171535_0223 from job_202007171535_0223

  3. Run the below YARN cmd to get YARN job logs (common for hive/hadoop/spark jobs which are managed by YARN):

    yarn logs -applicationId <your app id here>

Upvotes: 2

Related Questions