CODEWITHSUNDEEP
scalaapache-sparksbthadoop-yarn
a.moussa
a.moussa

Reputation: 3277

java.lang.NoSuchMethodError: when spark-submit on yarn cluster

I have a spark application running correctly in local mode. when i run spark-submit on yarn cluster, I obtain the following error:

18/07/26 18:12:38 ERROR ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder;
java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder;
        at fr.test.ssl.SecrityHttpClient$.getHttpClientWithoutSSL(SecrityHttpClient.scala:23)
        at fr.test.processor.HttpProcessor$.execute(HttpProcessor.scala:36)
        at fr.test.engine.RequestEngine$$anonfun$executeHttpRequest$2.apply(RequestEngine.scala:28)
        at fr.test.engine.RequestEngine$$anonfun$executeHttpRequest$2.apply(RequestEngine.scala:21)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at fr.test.engine.RequestEngine$.executeHttpRequest(RequestEngine.scala:21)
        at fr.test.launcher.Launcher$.executeRequestList(Launcher.scala:20)
        at fr.test.launcher.Launcher$.main(Launcher.scala:10)
        at fr.test.launcher.Launcher.main(Launcher.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:559)
18/07/26 18:12:38 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder;)

It seems httpclient dependency is not find . Here is my build 

import aether.AetherKeys._

name := "my_app"

organization := "fr.test"
version := "0.1"
scalaVersion := "2.10.6"

val httpclientVersion = "4.5.6"
val slickVersion = "3.1.1"
val hikariCPVersion = "2.4.6"

libraryDependencies += "com.google.code.gson" % "gson" % "2.8.5"
libraryDependencies += "com.typesafe.slick" %% "slick-hikaricp" % slickVersion exclude("com.zaxxer", "HikariCP-java6")
libraryDependencies += "com.zaxxer" % "HikariCP" % hikariCPVersion

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.2"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.6.2"
libraryDependencies += "com.springml" %% "spark-sftp" % "1.0.2"
// logging
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.7"
libraryDependencies += "com.sndyuk" % "logback-more-appenders" % "1.4.2"

// https://mvnrepository.com/artifact/com.databricks/spark-csv
libraryDependencies += "com.databricks" %% "spark-csv" % "1.5.0"

libraryDependencies += "org.apache.httpcomponents" % "httpclient" % httpclientVersion

libraryDependencies += "org.postgresql" % "postgresql" % "9.4.1208"

aetherOldVersionMethod := true
overridePublishSettings

mainClass in assembly := Some("fr.test.Launcher")
assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case "reference.conf" => MergeStrategy.concat
  case x => MergeStrategy.first
}

// Configure assembly artifact to be published
artifact in(Compile, assembly) := {
  val art = (artifact in(Compile, assembly)).value
  art.withClassifier(Some("assembly"))
}
addArtifact(artifact in(Compile, assembly), assembly)

I opened the fat jar generated by assembly command and I find the dependency inside the jar so I don't understand why I obtain this error. We use the correct scala and spark version according to scala and spark version intalled on the cluster. here is my spark-submit:

spark-submit --class fr.test.Launcher \
--master yarn-cluster \
--num-executors 4 \
--driver-memory 10g \
--executor-memory 5g \
--queue dlk_dev \
--files /home/my_user/my_app_2.10/-SNAPSHOT/application--SNAPSHOT.conf#app.conf \
--conf "spark.driver.extraJavaOptions=-verbose:class" \
--conf "spark.executor.extraJavaOptions=-verbose:class" \
/home/my_user/my_app_2.10/-SNAPSHOT/my_app_2.10--SNAPSHOT.jar

I also find this line searching in yarn logs

[Loaded org.apache.http.impl.client.HttpClientBuilder from file:/data/5/yarn/local/filecache/216/spark-hdp-assembly.jar]

Do you have any idea?

Upvotes: 0

Views: 3020

Answers (1)

Kevin Mooney
Kevin Mooney

Reputation: 66

spark comes with its own http-client. Likely you are using a different version of http-client than what is deployed in your yarn cluster. You can set the spark config option spark.executor.userClassPathFirst to true to have your user provided jars (well just your uber jar in this case) placed first on the classpath. This should allow your version of httpclient to be picked up first.

Upvotes: 2

Related Questions