CODEWITHSUNDEEP
scaladatabricks
sutugin
sutugin

Reputation: 93

problems with use databrics dbutils-api

How do we try use it?

When we package the code into a jar we get:

[error] missing or invalid dependency detected while loading class file 'NotebookUtils.class'.
[error] Could not access term common in package com.databricks.backend,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'NotebookUtils.class' was compiled against an incompatible version of com.databricks.backend.
[error] one error found
[error] (compile:compileIncremental) Compilation failed

Upvotes: 5

Views: 1412

Answers (1)

morsik
morsik

Reputation: 1300

The reason is even latest ver of dbutils-api for now (which is 0.0.3) relies on com.databricks.backend.common (which is neither not part of dbutils-api nor any other public databricks libs).

trait NotebookUtils ... {
...
  def getContext() : com.databricks.backend.common.rpc.CommandContext
  def setContext(ctx : com.databricks.backend.common.rpc.CommandContext) : scala.Unit
}

While:

jar tvf dbutils-api_2.11-0.0.3.jar | grep backend
   439 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/Default.class
  1337 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/FileInfo$$anonfun$hashCode$1.class
  1265 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/FileInfo$$anonfun$hashCode$2.class
  4109 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/FileInfo.class
  1256 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/GrantInfo.class
  1480 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/MountInfo.class
   452 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/PackageGroup.class
     0 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/
     0 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/
     0 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/

The most probable reason is com.databricks.backend.common marked with provided scope in dbutils definition file (but I can't be 100% sure because this lib is not available in open source). I would rather be intended to treat this as a bug in Databricks' libs.

Upvotes: 1

Related Questions