CODEWITHSUNDEEP
scalaapache-sparkversionhadoop-yarnmapr
MapReddy Usthili
MapReddy Usthili

Reputation: 288

java.io.InvalidClassException: scala.reflect.ClassTag$$anon$1; local class incompatible:: Spark

I'm getting below exception while counting the file rows : Actually code is working fine if i executes shell in local mode : But If we try shell with Yarn-client mode -- the below exceptions stack will be returning. I've been investigated a little bit for the same problem. some are suggesting it's different JVM compatibility issue . It might be true because in other environment am using JDK6 it is working perfect in both local and Yarn-Client modes . In Another Environment am using JDK7 . here am facing the problem while running in yarn-client mode : 

java.io.InvalidClassException: scala.reflect.ClassTag$$anon$1; local class incompatible: stream classdesc serialVersionUID = -4937928798201944954, local class serialVersionUID = -8102093212602380348

FYI JAVA - :In Environment am facing the problem 

java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470_27sr2fp10-20141218_02(SR2 FP10))
IBM J9 VM (build 2.7, JRE 1.7.0 Linux amd64-64 Compressed References 20141215_227395 (JIT enabled, AOT enabled)
J9VM - R27_Java727_SR2_20141215_1631_B227395
JIT  - tr.r13.java_20141003_74587.07
GC   - R27_Java727_SR2_20141215_1631_B227395_CMPRSS
J9CL - 20141215_227395)
JCL - 20141217_01 based on Oracle jdk7u75-b12

In another old Environment : it is working fine 

java -version
java version "1.6.0_91"
Java(TM) SE Runtime Environment (build 1.6.0_91-b31)
Java HotSpot(TM) 64-Bit Server VM (build 20.91-b07, mixed mode

From here I had moved the code to new Environment : If we suspects that it is incompatible version Issue then it shouldn't work in local mode too. Please clarify my questions:

Upvotes: 0

Views: 645

Answers (2)

Ram Ghadiyaram
Ram Ghadiyaram

Reputation: 29195

I also experienced the same issue, my cluster using RHEL and jre-1.7.1-ibm.x86_64.

After spending many hours, I found that the cause of the problem came from the JVM

Solution

I was able to solve this issue by changing the jvm from jre-1.7.1-ibm.x86_64

to java-1.8.0-openjdk-1.8.0.25-3.b17.el6_6.x86_64.

Upvotes: 0

MapReddy Usthili
MapReddy Usthili

Reputation: 288

I fixed it By after setting JAVA_HOME to 1.8 export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.31-1.b13.el6_6.x86_64/ export PATH=$JAVA_HOME/bin:$PATH

Upvotes: 1

Related Questions