Maven - Different Dependency Version in Test

Question

I'm suffering an issue similar to Maven 2 - different dependency versions in test and compile but the specified answer there does not work.

In my project I need to depend on a Cloudera distribution of Hadoop and a 'vanilla' version for JUnit testing, as the former only works on *nix.

When I try and execute my application, I get Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration. When I run JUnit tests from Maven or Eclipse, everything works fine. If I comment out the test dependencies, the application runs successfully.

Why is the compile dependency getting ignored when the test dependency is uncommented?

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>0.20.2-cdh3u2</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.0.0</version>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-test</artifactId>
        <version>1.0.0</version>
        <scope>test</scope>
    </dependency>

mvn dependency:list is shown below. The compile scoped version does not show up at all:

[INFO] The following files have been resolved:
[INFO]    ant:ant:jar:1.6.5:test
[INFO]    aopalliance:aopalliance:jar:1.0:compile
[INFO]    asm:asm:jar:3.3.1:compile
[INFO]    cglib:cglib:jar:2.2.2:compile
[INFO]    ch.qos.logback:logback-classic:jar:1.0.0:compile
[INFO]    ch.qos.logback:logback-core:jar:1.0.0:compile
[INFO]    com.google.guava:guava:jar:r08:compile
[INFO]    com.h2database:h2:jar:1.3.164:test
[INFO]    com.jolbox:bonecp:jar:0.7.1.RELEASE:compile
[INFO]    com.sun.jersey:jersey-core:jar:1.11:test
[INFO]    commons-beanutils:commons-beanutils:jar:1.7.0:test
[INFO]    commons-beanutils:commons-beanutils-core:jar:1.8.0:test
[INFO]    commons-cli:commons-cli:jar:1.2:test
[INFO]    commons-codec:commons-codec:jar:1.4:test
[INFO]    commons-collections:commons-collections:jar:3.2.1:test
[INFO]    commons-configuration:commons-configuration:jar:1.6:test
[INFO]    commons-digester:commons-digester:jar:1.8:test
[INFO]    commons-el:commons-el:jar:1.0:test
[INFO]    commons-httpclient:commons-httpclient:jar:3.0.1:test
[INFO]    commons-lang:commons-lang:jar:2.4:test
[INFO]    commons-logging:commons-logging:jar:1.1.1:compile
[INFO]    commons-net:commons-net:jar:1.4.1:test
[INFO]    hsqldb:hsqldb:jar:1.8.0.10:test
[INFO]    junit:junit:jar:4.10:test
[INFO]    mysql:mysql-connector-java:jar:5.1.18:compile
[INFO]    net.java.dev.jets3t:jets3t:jar:0.7.1:test
[INFO]    net.sf.kosmosfs:kfs:jar:0.3:test
[INFO]    org.apache.commons:commons-math:jar:2.1:test
[INFO]    org.apache.ftpserver:ftplet-api:jar:1.0.0:test
[INFO]    org.apache.ftpserver:ftpserver-core:jar:1.0.0:test
[INFO]    org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:test
[INFO]    org.apache.hadoop:hadoop-core:jar:1.0.0:test
[INFO]    org.apache.hadoop:hadoop-test:jar:1.0.0:test
[INFO]    org.apache.mina:mina-core:jar:2.0.0-M5:test
[INFO]    org.codehaus.jackson:jackson-core-asl:jar:1.0.1:test
[INFO]    org.codehaus.jackson:jackson-mapper-asl:jar:1.0.1:test
[INFO]    org.eclipse.jdt:core:jar:3.1.1:test
[INFO]    org.hamcrest:hamcrest-core:jar:1.1:test
[INFO]    org.liquibase:liquibase-core:jar:2.0.3:test
[INFO]    org.liquibase.ext:liquibase-slf4j:jar:0.0.1:test
[INFO]    org.mortbay.jetty:jetty:jar:6.1.26:test
[INFO]    org.mortbay.jetty:jetty-util:jar:6.1.26:test
[INFO]    org.mortbay.jetty:jsp-2.1:jar:6.1.14:test
[INFO]    org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:test
[INFO]    org.mortbay.jetty:servlet-api:jar:2.5-20081211:test
[INFO]    org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:test
[INFO]    org.slf4j:jcl-over-slf4j:jar:1.6.4:compile
[INFO]    org.slf4j:log4j-over-slf4j:jar:1.6.4:compile
[INFO]    org.slf4j:slf4j-api:jar:1.6.4:compile
[INFO]    org.springframework:spring-aop:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-asm:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-beans:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-context:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-context-support:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-core:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-expression:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-jdbc:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-test:jar:3.1.1.RELEASE:test
[INFO]    org.springframework:spring-tx:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework.data:spring-data-hadoop:jar:1.0.0.BUILD-SNAPSHOT:c
ompile
[INFO]    oro:oro:jar:2.0.8:test
[INFO]    tomcat:jasper-compiler:jar:5.5.12:test
[INFO]    tomcat:jasper-runtime:jar:5.5.12:test
[INFO]    xmlenc:xmlenc:jar:0.52:test

Answer 1

What you desire, is no longer valid for Maven 3 (this used to be valid for Maven 2). Maven 3 will attempt to obtain the nearest dependency, effectively ensuring that only one of the compile or test scoped dependency is used for both the compile and test phases.

In your case org.apache.hadoop:hadoop-core:1.0.0:test overrides org.apache.hadoop:hadoop-core:0.20.2-cdh3u2:compile and hence becomes the nearest dependency. You may see the following warning displayed by Maven when running the mvn dependency:list goal, that alludes to this being a problem in your project model:

[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-core:jar -> version 0.20.2-cdh3u2 vs 1.0.0 @ line xyz, column xyz

To 'fix' this, it would be better to split your tests into a separate project whose project model can define a separate set of test dependencies.

Answer 2

Looks like a maven bug - or if not, something that is fishy enough I wouldn't trust the documentation if there were any.

In your case, I would probably:

1. upgrade maven and see if the issue is fixed in the latest m3
2. try repackaging one of the hadoop versions with another artifact id or group name. maven-shade-plugin, don't bother changing the package name, but just introduce something maven won't know is no different from org.apache.hadoop:hadoop-core
3. try moving the tests to a separate project downstream
4. start debugging maven and understand what's going on exactly (and then fall back on 1 or 2...)