CODEWITHSUNDEEP
mergesbtscalatestsbt-assembly
user1187135
user1187135

Reputation:

Fail to override Assembly Merge Strategy in multi-jvm node test

I am having trouble overriding the merge strategy for a dependency.

The problem stems from the cassandra dependency rely on separate netty modules 

val akkaCassandra = "com.typesafe.akka" %% "akka-persistence-cassandra" % "0.17"

If I run the graph-dependency, it outputs:

  | +-io.netty:netty-handler:4.0.33.Final
[info]     |   +-io.netty:netty-buffer:4.0.33.Final
[info]     |   | +-io.netty:netty-common:4.0.33.Final
[info]     |   |
[info]     |   +-io.netty:netty-codec:4.0.33.Final
[info]     |   | +-io.netty:netty-transport:4.0.33.Final
[info]     |   |   +-io.netty:netty-buffer:4.0.33.Final
[info]     |   |     +-io.netty:netty-common:4.0.33.Final
[info]     |   |
[info]     |   +-io.netty:netty-transport:4.0.33.Final
[info]     |     +-io.netty:netty-buffer:4.0.33.Final
[info]     |       +-io.netty:netty-common:4.0.33.Final

According to this discussion, its best to just define a merge strategy as a solution:

https://groups.google.com/a/lists.datastax.com/forum/#!msg/spark-connector-user/5muNwRaCJnU/sIHYh6PFEwAJ

Netty unfortunately has timing dependent markers in each io.versions.properties file in the metainfUntitled.jpg

This means the various components included all have different timestamps which is why everything is breaking. Unfortunately this is the underlying C* driver's dep, we could I guess exclude these netty modules from the driver and include netty-all in the connector instead but that seems like overkill.

I think just fixing the application build file is still the best solution.

According to this, the ability to override should be implemented:

https://github.com/typesafehub/sbt-multi-jvm/issues/22

Nested assembly config should be exposed for customization along with the other multijvm keys so that users can solve these issues.

i think that PR #19 will enable this functionality. the change at msfrank@fe862ff#diff-ad54d47177586fbaf474e402dd1b3dc5R137 will pass through the merge strategy defined with settings key (assemblyMergeStrategy in assembly) as long as the file is not one of *.class, *.txt, or NOTICE, which are hardcoded merge strategies

.

But when I run

sbt:multi-node-test

lazy val test = Project(id = "core-tests", base = file("./modules/core/tests"))
  .settings(SbtMultiJvm.multiJvmSettings: _*)
  .settings(
    libraryDependencies ++= Dependencies.coreTests,
    assemblyMergeStrategy in assembly := {
      case x if x.endsWith("META-INF/io.netty.versions.properties") ⇒ MergeStrategy.first
    },
    compile in MultiJvm <<= (compile in MultiJvm) triggeredBy (compile in Test),
    parallelExecution in Test := false,
    executeTests in Test <<= (executeTests in Test, executeTests in MultiJvm) map {
      case (testResults, multiNodeResults) ⇒
        val overall =
          if (testResults.overall.id < multiNodeResults.overall.id)
            multiNodeResults.overall
          else
            testResults.overall
        Tests.Output(overall,
          testResults.events ++ multiNodeResults.events,
          testResults.summaries ++ multiNodeResults.summaries)
    },
    licenses := Seq(("CC0", url("http://creativecommons.org/publicdomain/zero/1.0"))),
    Settings.levelDb, Settings.test)

it still refused to use the merge strategy, I defined. 

[error] 1 error was encountered during merge

[trace] Stack trace suppressed: run 'last core-tests/multi-jvm:assembly' for the full output.

[error] (core-tests/multi-jvm:assembly) deduplicate: different file contents found in the following:

[error] ~/.ivy2/cache/io.netty/netty-handler/jars/netty-handler-4.0.33.Final.jar:META-INF/io.netty.versions.properties

[error] ~/.ivy2/cache/io.netty/netty-buffer/jars/netty-buffer-4.0.33.Final.jar:META-INF/io.netty.versions.properties

[error] ~/.ivy2/cache/io.netty/netty-common/jars/netty-common-4.0.33.Final.jar:META-INF/io.netty.versions.properties

[error] ~/.ivy2/cache/io.netty/netty-transport/jars/netty-transport-4.0.33.Final.jar:META-INF/io.netty.versions.properties

[error] ~/.ivy2/cache/io.netty/netty-codec/jars/netty-codec-4.0.33.Final.jar:META-INF/io.netty.versions.propertie

Upvotes: 3

Views: 400

Answers (2)

y_ug
y_ug

Reputation: 1124

These files have different content, I see no much harm in concatenating them:

assemblyMergeStrategy in assembly  := {
  case PathList("META-INF", "io.netty.versions.properties") =>
    MergeStrategy.concat
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

Then see content:

unzip -p YOUR_ASSEBMBLY.jar "META-INF/io.netty.versions.properties"

Upvotes: 0

nikelin
nikelin

Reputation: 61

It doesn't work for you, because in your code you override settings for the sbt-assembly task.

What you need to do instead, is to override settings for the sbt-assembly in context of sbt-multi-jvm plugin.

You can do it this way:

assemblyMergeStrategy in assembly in MultiJvm := {
  case x if x.endsWith("META-INF/io.netty.versions.properties") ⇒ MergeStrategy.first
}

Upvotes: 1

Related Questions