In Maven, what is the difference between `package:artifact:jar:version` and package:artifact:jar:tests:version`?

Question

Using Maven 3.0.5

I'm trying to get spark-testing-base from com.holdenkarau to work with Hadoop 3.1. holdenkarau's dependency tree includes Hadoop 2.8.3; which is why I think I'm getting errors.

From my mvn dependency:tree I see the following lines:

[INFO] +- org.apache.hadoop:hadoop-common:jar:3.1.0:provided

...

[INFO] |  +- org.apache.hadoop:hadoop-common:jar:tests:2.8.3:test

These lines come from these two lines in the pom.xml file:

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>3.1.0</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>com.holdenkarau</groupId>
        <artifactId>spark-testing-base_${scala.compat.version}</artifactId>
        <version>${spark.version}_0.12.0</version>
        <scope>test</scope>
        <exclusions>
            <exclusion>
                <groupId>commons-beanutils</groupId>
                <artifactId>commons-beanutils-core</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

I basically have two related questions:

• What is the difference between org.apache.hadoop:hadoop-common:jar:3.1.0 and org.apache.hadoop:hadoop-common:jar:tests:2.8.3. What is that extra tests in there for; where does it come from and what does it mean?

• If I have a dependency that uses an older version of a package in the test scope, how do I force it to use a newer version; i.e., how do I force spark-testing-base to use Hadoop 3.1 in the test scope.

Answer 1

tests is called a classifier, and it contains code that's really only useful in the context of actually testing, such as an embedded HDFS system

You could explicitly try pulling in a new version like so, assuming it exists

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>3.1.0</version>
    <scope>test</scope>
     <classifier>test</classifier>
</dependency>

You may also want to exclude the same within the other dependency, however you might then run into build issues since that library is only written to test against 2.8.3