CODEWITHSUNDEEP
apache-sparkazure-databricksdatabricks-connect
zaxme
zaxme

Reputation: 1095

Databricks Connect: DependencyCheckWarning: The java class may not be present on the remote cluster

I was performing yet another execution of local Scala code against the remote Spark cluster on Databricks and got this.

Exception in thread "main" com.databricks.service.DependencyCheckWarning: The java class <something> may not be present on the remote cluster. It can be found in <something>/target/scala-2.11/classes. To resolve this, package the classes in <something>/target/scala-2.11/classes into a jar file and then call sc.addJar() on the package jar. You can disable this check by setting the SQL conf spark.databricks.service.client.checkDeps=false.

I have tried reimporting, cleaning and recompiling the sbt project to no avail.

Anyone know how to deal with this?

Upvotes: 3

Views: 950

Answers (1)

zaxme
zaxme

Reputation: 1095

Apparently the documentation has that covered:

spark.sparkContext.addJar("./target/scala-2.11/hello-world_2.11-1.0.jar")

I guess it makes sense that everything that you are writing as code external to Spark is considered a dependency. So a simple sbt publishLocal and then pointing to the jar path in above command will sort you out.

My main confusion came from the fact that I didn't need to do this for a very long while until at some point this mechanism kicked it. Rather inconsistent behavior I'd say.

A personal observation after working with this setup is that it seems you only need to publish a jar a single time. I have been changing my code multiple times and changes are reflected even though I have not been continuously publishing jars for the new changes I made. That makes the whole task a one off. Still confusing though.

Upvotes: 3

Related Questions