CODEWITHSUNDEEP
apache-sparkhadoopapache-spark-sql
BdEngineer
BdEngineer

Reputation: 3209

Spark Structure Streaming job failing in cluster mode

I am using spark-sql-2.4.1 v in my application.

While writing data on to hdfs folder I am facing this issue in spark-streaming application

Error:

    yarn.Client: Deleted staging directory hdfs://dev/user/xyz/.sparkStaging/application_1575699597805_47
    20/02/24 14:02:15 ERROR yarn.Client: Application diagnostics message: User class threw exception: org.apache.hadoop.security.AccessControlException: Permission denied: user= xyz, access=WRITE, inode="/tmp/hadoop-admin":admin:supergroup:drwxr-xr-x
    .
    .
    .
    Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=xyz, access=WRITE, inode="/tmp/hadoop-admin":admin:supergroup:drwxr-xr-x
            at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:350)
            at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:251)

While writing data on to HDFS folder I am facing this issue in spark-streaming application. When I run in yarn-cluster mode I face this issue i.e.

    --master yarn \
    --deploy-mode cluster \

But when I run in “yarn-client” mode it runs fine i.e.

    --master yarn \
    --deploy-mode client \

What is the root cause of this problem?

Fundamental question here, why it is trying to write in "/tmp/hadoop-admin/" instead of respective user directory i.e. hdfs://qa2/user/xyz/?

I have come across this fix:

https://issues.apache.org/jira/browse/SPARK-26825

How can I implement it in my spark-sql application?

Upvotes: 0

Views: 838

Answers (1)

Jacek Laskowski
Jacek Laskowski

Reputation: 74779

The only difference between the working --deploy-mode client and the failing --deploy-mode cluster cases is the location of the driver. In client deploy mode, the driver runs on the machine you execute spark-submit (which is usually an edge node that is configured to use a YARN cluster, but it is not part of it) while in cluster deploy mode the driver runs as part of a YARN cluster (one of the nodes under control of YARN).

It looks like you've got a misconfigured edge node.

I'd not be surprised if a regular Spark SQL-only Spark application would be failing too. I'd not be surprised to hear that it has nothing to do with a streaming query (Spark Structured Streaming) and would fail for any Spark application.

Upvotes: 1

Related Questions