CODEWITHSUNDEEP
spark-submit
zhicheng
zhicheng

Reputation: 184

java.lang.IllegalArgumentException: Too large frame: 5211883372140375593

I submitted my code to the cluster to run, but I encountered the following error.

''' java.lang.IllegalArgumentException: Too large frame: 5211883372140375593 at org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)

'''

and my submit command is like this

spark-submit
--master spark://172.16.244.8:6066
--deploy-mode cluster
--num-executors 3
--executor-cores 8
--executor-memory 16g
--driver-memory 2g
--conf spark.default.parallelism=10
--class ParallelComputing
hdfs://172.16.244.5:9000/qlt/portrait-batch-0.0.1-jar-with-dependencies.jar

what is the reason

Upvotes: 9

Views: 9706

Answers (2)

Ke Liu
Ke Liu

Reputation: 11

In my spark project I face this same issue ,I found my spark version is 3.3.1 and hadoop version(hadoop-client-runtime) is 3.3.2. So I change my hadoop-client-runtime version from 3.3.2 to 3.3.1. After I tested, the problem was solved.

Upvotes: 1

zhicheng
zhicheng

Reputation: 184

The reason is because the version of spark does not match the version of the cluster, which can be solved by modifying the local spark version to be consistent with the cluster.

Upvotes: 4

Related Questions