CODEWITHSUNDEEP
scalahadoopsbt
Kadir Gunel
Kadir Gunel

Reputation: 331

intransitive dependencies with sbt assembly

I downloaded a (active) git project which requires me to install scala(sbt too) and hadoop. When I try to built it via sbt (sbt assembly) it produces tons of warnings about intransitive dependencies. I tried to contact with the project suppliers but they do not give me proper responses; just telling me that it is working on their machines.

When I try "sbt assembly" I get :

[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[warn] Found intransitive dependency (org.apache.hadoop:hadoop-common:2.6.0) while publishMavenStyle is true, but Maven repositories
[warn]   do not support intransitive dependencies. Use exclusions instead so transitive dependencies
[warn]   will be correctly excluded in dependent projects.
[warn]            
[warn] Found intransitive dependency (org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0) while publishMavenStyle is true, but Maven repositories
[warn]   do not support intransitive dependencies. Use exclusions instead so transitive dependencies
[warn]   will be correctly excluded in dependent projects.
[warn]            
[warn] Found intransitive dependency (org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0) while publishMavenStyle is true, but Maven repositories
[warn]   do not support intransitive dependencies. Use exclusions instead so transitive dependencies
[warn]   will be correctly excluded in dependent projects.
[warn]            
[warn] Found intransitive dependency (org.apache.hadoop:hadoop-annotations:2.6.0) while publishMavenStyle is true, but Maven repositories
[warn]   do not support intransitive dependencies. Use exclusions instead so transitive dependencies
[warn]   will be correctly excluded in dependent projects.
[warn]            
[warn] Found intransitive dependency (org.apache.hadoop:hadoop-auth:2.6.0) while publishMavenStyle is true, but Maven repositories
[warn]   do not support intransitive dependencies. Use exclusions instead so transitive dependencies
[warn]   will be correctly excluded in dependent projects.
[warn]            
[warn] Found intransitive dependency (org.apache.hadoop:hadoop-yarn-common:2.6.0) while publishMavenStyle is true, but Maven repositories
[warn]   do not support intransitive dependencies. Use exclusions instead so transitive dependencies
[warn]   will be correctly excluded in dependent projects.
[warn]            
[warn] Found intransitive dependency (org.apache.hbase:hbase-server:0.96.2-hadoop2) while publishMavenStyle is true, but Maven repositories
[warn]   do not support intransitive dependencies. Use exclusions instead so transitive dependencies
[warn]   will be correctly excluded in dependent projects ...

And there is a build.sbt file which contains all the dependencies that project needs: 

libraryDependencies ++= Seq(
            "com.beust" % "jcommander" % "1.35",
            "org.apache.hadoop" % "hadoop-common" % "2.6.0" intransitive(),
            "org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.6.0"  intransitive(),
            "org.apache.hadoop" % "hadoop-mapreduce-client-common" % "2.6.0"  intransitive(),
            "org.apache.hadoop" % "hadoop-annotations" % "2.6.0" intransitive(), 
            "org.apache.hadoop" % "hadoop-auth" % "2.6.0" intransitive(),
            "org.apache.hadoop" % "hadoop-yarn-common" % "2.6.0" intransitive(),
            "org.apache.hadoop" % "hadoop-yarn-api" % "2.6.0",
            "org.apache.hbase" % "hbase-server" % "0.96.2-hadoop2" intransitive(), 
            "org.apache.hbase" % "hbase-common" % "0.96.2-hadoop2"  intransitive(), 
            "org.apache.hbase" % "hbase-client" % "0.96.2-hadoop2" intransitive(),
            "org.apache.hbase" % "hbase-protocol" % "0.96.2-hadoop2",
            "org.cloudera.htrace" % "htrace-core" % "2.04" intransitive(),
            "commons-configuration" % "commons-configuration" % "1.6" exclude("commons-beanutils", "commons-beanutils-core"),
            "commons-httpclient" % "commons-httpclient" % "3.1",
            "commons-io" % "commons-io" % "2.4",
            "com.google.guava" % "guava" % "11.0.2",
            "log4j" % "log4j" % "1.2.16",
            "org.slf4j" % "slf4j-log4j12" % "1.7.5",
            "org.codehaus.jackson" % "jackson-jaxrs" % "1.9.13",
            "org.apache.avro" % "avro" % "1.7.4",
            "junit" % "junit" % "4.11" % "it,test",
            "com.novocode" % "junit-interface" % "0.10" % "it,test",
            "com.jsuereth" % "scala-arm_2.11" % "1.4" % "it,test",
            "org.scalatest" % "scalatest_2.11" % "2.2.1" % "it,test"
            )

assemblyMergeStrategy in assembly := {
    case PathList("org", "apache", "hadoop", "yarn", xs @ _*) => MergeStrategy.first 
    case x => val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

assemblyExcludedJars in assembly := { 
  val cp = (fullClasspath in assembly).value
  cp filter {_.data.getName == "commons-beanutils-1.7.0.jar"}
}

scalaVersion := "2.11.4"

// output jar is here: target/ruleXtract.jar
assemblyOutputPath in assembly := file("target/ruleXtract.jar") 

// we want a jar without a main class so we can run it as "hadoop jar class args"
mainClass in (Compile, packageBin) := None

What should I do in order to make it work? I have no idea how to build this project without "sbt assembly". I checked some tutorials about sbt and the build.sbt seems fine - though not really sure. What is the problem? Do you have any suggestions to solve it ?

B.R.

Upvotes: 2

Views: 1584

Answers (1)

Greg
Greg

Reputation: 752

This may not solve your problem, but it'll get rid of the warnings. The SBT property 'publishMavenStyle' is used when you want to publish your project to a Maven repository (http://xerial.org/blog/2014/03/24/sbt/). If you're not planning on publishing, you can set it to false.

If you're using multiple build.sbt files, you can set it as such, which will set it for the entire project:

publishMavenStyle in ThisBuild := false

Upvotes: 2

Related Questions