CODEWITHSUNDEEP
scalasbt
user3825558
user3825558

Reputation: 585

Scala: SBT: mergeStrategy: deduplicate: different file contents found in the following:

I am trying to build a fat Jar using the sbt-assembly plugin. I am on SBT version 0.13.1 and assembly plugin version: 0.11.2. Whenever I run the assembly command in the sbt console I am encountering this error:

Update: 

    [warn] Merging 'META-INF\LICENSE' with strategy 'rename'
    [warn] Merging 'META-INF\DEPENDENCIES' with strategy 'discard'
    [warn] Merging 'META-INF\MANIFEST.MF' with strategy 'discard'
    [debug] Merging 'META-INF\native\linux32\libleveldbjni.so' with strategy 'deduplicate'
    [debug] Merging 'META-INF\native\linux64\libleveldbjni.so' with strategy 'deduplicate'
    [debug] Merging 'META-INF\native\osx\libleveldbjni.jnilib' with strategy 'deduplicate'
    [debug] Merging 'META-INF\native\windows32\leveldbjni.dll' with strategy 'deduplicate'
    [debug] Merging 'META-INF\native\windows64\leveldbjni.dll' with strategy 'deduplicate'
    [debug] Merging 'default.key' with strategy 'deduplicate'
    java.lang.RuntimeException: deduplicate: different file contents found in the following:
    mydef.key
    mydef.key
            at sbtassembly.Plugin$Assembly$.sbtassembly$Plugin$Assembly$$applyStrategy$1(Plugin.scala:253)
            at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:270)
            at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:267)
            at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
            at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
            at scala.collection.Iterator$class.foreach(Iterator.scala:727)
            at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
            at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
            at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
            at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
            at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
            at sbtassembly.Plugin$Assembly$.applyStrategies(Plugin.scala:272)
            at sbtassembly.Plugin$Assembly$.x$4$lzycompute$1(Plugin.scala:172)
            at sbtassembly.Plugin$Assembly$.x$4$1(Plugin.scala:170)
            at sbtassembly.Plugin$Assembly$.stratMapping$lzycompute$1(Plugin.scala:170)
            at sbtassembly.Plugin$Assembly$.stratMapping$1(Plugin.scala:170)
            at sbtassembly.Plugin$Assembly$.inputs$lzycompute$1(Plugin.scala:214)
            at sbtassembly.Plugin$Assembly$.inputs$1(Plugin.scala:204)
            at sbtassembly.Plugin$Assembly$.apply(Plugin.scala:230)
            at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:373)
            at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:370)
            at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
            at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
            at sbt.std.Transform$$anon$4.work(System.scala:64)
            at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
            at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
            at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
            at sbt.Execute.work(Execute.scala:244)
            at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
            at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
            at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
            at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)
    [error] (cloud_ui/*:assembly) deduplicate: different file contents found in the following:
    [error] mydef.key
    [error] mydef.key

**[error](mymodule:*.assembly) deduplicate: different file contents found in the following:
[error] mydef.key
[error] mydef.key**


On inspecting my build.sbt I find that IntelliJ complains about "mergeStrategy in assembly" block. Its complaint is: Type mismatch expected: Def Initialize[(String) => Plugin.MergeStrategy], actual: DefInitialize[Any]

Here is my mergeStrategy: 

       mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
             {
                case PathList("org", "jboss", "netty", xs @ _*)  =>  MergeStrategy.first
      case PathList("META-INF", "jboss-beans.xml")         => MergeStrategy.first
      case PathList("org", "w3c", "dom", "TypeInfo.class")  => MergeStrategy.first
      case "jboss-beans.xml" => MergeStrategy.first
      case "application.conf" => MergeStrategy.concat
      case "unwanted.txt"     => MergeStrategy.discard
      case "rootdoc.txt"     => MergeStrategy.discard
      case x => old(x)
    }
   }

I have looked up some of the resources on stackoverflow that address a similar problem and I am wondering if the following resource has the solution to my problem.

From a knowledge perspective, I have learned just enough to use SBT for my build purposes, but I have not got been able to delve into its murky depths so far. But I am ready to work on it. If someone can point me in the right direction I can go off and correct the error in the build.sbt file and also my 'deduplicate error"so that I can accomplish my goal of building a fat jar that I can deploy..

Update on what I tried myself, but failed to get it work: I had modified my mergeStrategy to include this line:

case PathList(ps @ _*) if ps.last endsWith ".key" => MergeStrategy.first

Modified mergeStrategy now looks like: 

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => 
 {
  case PathList(ps @ _*) if ps.last endsWith ".key" => MergeStrategy.first
  case PathList("org", "jboss", "netty", xs @ _*)         => MergeStrategy.first
  case PathList("META-INF", "jboss-beans.xml")         => MergeStrategy.first
  case PathList("org", "w3c", "dom", "TypeInfo.class")         => MergeStrategy.first
  case "jboss-beans.xml" => MergeStrategy.first
  case "application.conf" => MergeStrategy.concat
  case "unwanted.txt"     => MergeStrategy.discard
  case "rootdoc.txt"     => MergeStrategy.discard
  case x =>
    //val old = (mergeStrategy in assembly).value
       old(x)
   }
}

Update: I solved the problem. I simply deleted the extra files and rebuilt again. And I did not have to do anything else to the merge-strategy.

Upvotes: 4

Views: 5621

Answers (1)

kpbochenek
kpbochenek

Reputation: 469

It looks like you got 2 files called mydef.key But because their content is not the same and sbt-assembly doesn't have rule how to merge it then exception is raised.

Check what are both contents and if they are important for your build. If not just try to add:

case "mydef.key" => MergeStrategy.first

Upvotes: 4

Related Questions