CODEWITHSUNDEEP
apache-sparkapache-commons-lang3
jb44
jb44

Reputation: 403

Spark 2.2 commons-lang3

I'm using the Spark 2.2 Snapshot (it includes a fix I need). I can run my application by firing up a "local" spark and everything is fine. When I run a job using spark-submit I get the following error:

17/04/19 13:25:42 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1, 192.168.13.231, executor 0): java.io.InvalidClassException:
org.apache.commons.lang3.time.FastDateParser; local class incompatible: stream classdesc serialVersionUID = 2, local class serialVersionUID = 3

There are some posts out there related to this issue, and these suggest the incompatibility arises from an older version of commons-lang3 on the classpath. But I've explicitly set the version of commons-lang3 (3.5), verified that the application only depends on 3.5 (using sbt-dependency-graph), and even removed all commons-lang3 files from the resulting jar. None of these has fixed the issue.

Any help much appreciated here - where would spark be picking up the older dependency if it's not in the submitted jar file? Am I misinterpreting the error message entirely?

Thanks

Upvotes: 0

Views: 1290

Answers (1)

jb44
jb44

Reputation: 403

Just in case this helps anyone - the jar conflict is coming from the alluxio client that has been added to spark. Although I haven't tried yet, I think squaring this version with the spark version will do the trick here.

Upvotes: 0

Related Questions