CODEWITHSUNDEEP
apache-sparksbtdruid
Ashish Awasthi
Ashish Awasthi

Reputation: 1327

Spark + Druid Tranquility - library version conflict

I get following error when I run a spark job with Druid Tranquility.

java.lang.NoSuchFieldError: WRITE_DURATIONS_AS_TIMESTAMPS

Druid Tranquility uses higher version of jackson-databind (2.6.1) than what is bundled in spark. I'm using latest stable versions of Druid Tranquility(0.6.4) and Spark(1.5.2).

How to resolve this?

  1. Force spark to use the classes supplied with the job and not the classes bundled in spark. How?
  2. Compile spark with the new version of jackson-databind
  3. Force Tranquility to use old version. (Doesn't seem to work, tried to exclude in sbt)
  4. Any other way?

Upvotes: 2

Views: 915

Answers (3)

Sunakshi Saxena
Sunakshi Saxena

Reputation: 1

add --jars hibernate-validator-5.1.3.Final.jar in the spark-submit, this may help

Upvotes: 0

Ashish Awasthi
Ashish Awasthi

Reputation: 1327

Gian has given a pull request https://github.com/druid-io/tranquility/pull/81 with downgrade Jackson to match the version in Druid. And announced that next version of tranquility will have it.

Upvotes: 0

Michael Spector
Michael Spector

Reputation: 37019

  1. You can force Spark to use classes supplied with the job using spark.driver.extraClassPath and spark.executor.extraClassPath configuration options, which will prepend anything you specify there to the classpath of Spark driver or Spark executor respectively.

  2. You can try to compile Spark against newer version of jackson-databind. To do that, update fasterxml.jackson.version configuration parameter in Spark's pom.xml file, then follow the Spark build instructions. It's not guaranteed that it will compile successfully though.

  3. Excluding jackson from your Tranquility application might work too, just make sure to exclude the following artifacts:

    com.fasterxml.jackson.core:jackson-core com.fasterxml.jackson.core:jackson-annotations com.fasterxml.jackson.dataformat:jackson-dataformat-smile com.fasterxml.jackson.datatype:jackson-datatype-joda com.fasterxml.jackson.core:jackson-databind

You can verify that these artifacts are excluded by running sbt dependency-tree (using this plug-in).

I'd first try (3) then (2) and (1).

Upvotes: 2

Related Questions