CODEWITHSUNDEEP
apache-sparkooziehuecloudera-manager
taransaini43
taransaini43

Reputation: 184

Spark job ( oozie - hue) stuck in running state

I have a spark job to convert csv to parquet. And I am trying to run it via oozie workflow in hue. It is the simplest workflow comprising a single step(a spark program).

When I run it for a simple program jar(say hello spark types example), it works fine when submitted via hue.

But when I have a bigger jar(~96 MB), the job gets stuck in running state. There is no issue with the code as the same jar works perfectly with spark-submit on the very same environment, with the same conditions - running both in client mode for simplicity of debugging.

Also, normally you can view from logs if there are certain exceptions or the job is hung due to memory issues(the continuous 'heart beat' info logs) but in this case I cannot even view the logs. When i manually kill the job, still no logs are accessible, it says :

Could not find job job_1481270830724_0007.

{"RemoteException":{"exception":"NotFoundException","message":"java.lang.Exception: job, job_1481270830724_0007, is not found","javaClassName":"org.apache.hadoop.yarn.webapp.NotFoundException"}} (error 404)

Can you help letting me know what could be the issue? Is it something to do with memory allocated for hue?

Upvotes: 0

Views: 786

Answers (1)

Mrgr8m4
Mrgr8m4

Reputation: 502

Which version of Hue are you using? I am asking cause there is a bug in Cloudera's JIRA. Better handling of expired YARN apps

Maybe you should upgrade version of Hue.

Upvotes: 0

Related Questions