CODEWITHSUNDEEP
hadoopmapreduceavro
jaksky
jaksky

Reputation: 3405

Avro mapreduce job failing java.lang.IncompatibleClassChangeError

I am using HDP 2.0 - hadoop 2.2.0 and Apache Avro 1.7.4 and followig avro mapreduce guide for a new M/R api here

I am getting following error

2014-07-15 05:46:04,354 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
    at org.apache.avro.mapreduce.AvroKeyInputFormat.createRecordReader(AvroKeyInputFormat.java:47)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:491)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

Seems that I am mixing old and new api. Any guess? Thx

Upvotes: 1

Views: 1558

Answers (2)

Don Smith
Don Smith

Reputation: 553

For me I solved the problem by adding classifier:"hapdoop2" to our build.gradle file:

 avro_mapred  : [group:"org.apache.avro", name:"avro-mapred", version:"1.7.7", classifier:"hadoop2"],

Upvotes: 0

jaksky
jaksky

Reputation: 3405

Actually found the solution to the problem. It is officially reported here

Simple solution - avro need to be compiled against hadoop 2

<dependency>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro-mapred</artifactId>
    <version>1.7.4</version>
    <classifier>hadoop2</classifier>
</dependency>

Upvotes: 2

Related Questions