CODEWITHSUNDEEP
hadoopmapreducecloudera
Capacytron
Capacytron

Reputation: 3729

java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.TaskAttemptID while running MRUnit

I'm trying to develop mr-job using Cloudera hadoop distrubution.I'm using api version 2. I do get trouble with mr-unit. Please advice what to do. I've used standard arhetype and completely lost, I don't get where is the rot of problem. Here are my dependencies:

<dependency>
            <groupId>com.cloudera.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>0.20.2-320</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>com.cloudera.hadoop</groupId>
            <artifactId>hadoop-mrunit</artifactId>
            <version>0.20.2-320</version>
            <scope>test</scope>
        </dependency>

Here is my test code:

@Test
    public void testEmptyOutput() throws Exception{
        for(String line : linesFromFlatFile){
            //List<Pair<GetReq, IntWritable>> output = 
                    driver.withInput(UNUSED_LONG_KEY, new Text(line) )
                 // .withOutput(null, null )
                  .run();
            //assertTrue("", output.isEmpty());
        }
    }

And here is an exception:

> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.283
> sec <<< FAILURE!
> testEmptyOutput(MapperTest)
> Time elapsed: 0.258 sec  <<< ERROR! java.lang.NoSuchMethodError:
> org.apache.hadoop.mapreduce.TaskAttemptID.<init>(Ljava/lang/String;IZII)V
>   at
> org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper$MockMapContext.<init>(MockMapContextWrapper.java:71)
>   at
> org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.getMockContext(MockMapContextWrapper.java:144)
>   at
> org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:197)
>   at
MapperTest.testEmptyOutput(ScoringCounterMapperTest.java:42)

package mypackage;

import java.util.Date;
import java.util.List;

import junit.framework.TestCase;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.apache.hadoop.mrunit.types.Pair;
import org.junit.Before;
import org.junit.Test;

import Sample;
import GetReq;



public class MapperTest extends TestCase {
    private static final IntWritable ONE_OCCURANCE = new IntWritable(1);
    private static final LongWritable UNUSED_LONG_KEY = new LongWritable(new Date().getTime());

    private Mapper<LongWritable, Text, GetReq, IntWritable> mapper;
    private MapDriver<LongWritable, Text, GetReq, IntWritable> driver;

    List<String> linesFromFlatFileNoOutput = null;
    List<String> linesFromFlatFileWithOutput = null;
    @Before 
    public void setUp() {
        mapper = newMapper();
        driver = new MapDriver<LongWritable, Text, GetReq, IntWritable>(mapper);
        Mapper.METADATA_CSV ="../../data/metadata.csv"; //ugly hook
        linesFromFlatFileNoOutput = Sample.instance.getLinesFromFlatFileNoOutput();  
        linesFromFlatFileWithOutput = Sample.instance.getLinesFromFlatFileWithOutput(); 
    }

    @Test
    public void testEmptyOutput() throws Exception{
        for(String line : linesFromFlatFileNoOutput){
            //List<Pair<GetReq, IntWritable>> output = 
                    driver.withInput(UNUSED_LONG_KEY, new Text(line) )
                  .withOutput(null, null )
                  .runTest();
            //assertTrue("", output.isEmpty());
        }
    }

    @Test
    public void testResultOutput() throws Exception{
        for(String line : linesFromFlatFileWithOutput){
            driver.withInput(UNUSED_LONG_KEY, new Text(line) )
                //.withOutput(null, null )
                .runTest();
        }
    }
}

Hm... I didn't change anything in pom.xml Now I get output and the same exeption. Looks like mapper runs once. Or rties to run. I get debug output from mapper body.

UPD: I've added classifier and changed dependency:

<dependency>
                <groupId>org.apache.mrunit</groupId>
                <artifactId>mrunit</artifactId>
                <version>0.9.0-incubating</version>
                <classifier>hadoop2</classifier>
                <scope>test</scope>
            </dependency>

Now I do get another problem:

Found interface org.apache.hadoop.mapreduce.Counter, but class was expected

on line:

context.getCounter(EnumCounter.MATCHED_RECORDS).increment(1);

What do I do wrong again?

Upvotes: 1

Views: 2497

Answers (2)

Capacytron
Capacytron

Reputation: 3729

I've found the solution: Nee to add classifier tag for mr-unit. And It should look like: 

<dependency> 
<groupId>org.apache.mrunit</groupId> 
<artifactId>mrunit</artifactId> 
<version>0.9.0-incubating</version> 
<classifier>hadoop2</classifier> 
<scope>test</scope> 
</dependency>

Now I have another problem: Found interface org.apache.hadoop.mapreduce.Counter, but class was expected on counter increment. This problem is related to some bug.

Upvotes: 1

Chris Gerken
Chris Gerken

Reputation: 16392

I've seen this kind of problem before when the wrong version of a dependency gets pulled into the runtime. There are two things that have fixed this kind of problem for me before:

  1. Change "provided" to "compile" on your hadoop-core dependency. This just forces the use at runtime of the right version. Go back to "provided" either way.
  2. In Eclipse, go to the pom editor, dependency hierarchy tab. You can see here if your choice of dependencies, taken together, refers to multiple versions of one of the indirect dependencies. If you find such a version, use a maven exclusion and an explicit dependency block to pick the one version you want to use.

Upvotes: 0

Related Questions