CODEWITHSUNDEEP
apache-sparkpysparkdatabricksazure-databrickswhitelist
lidong
lidong

Reputation: 608

Error running Spark on Databricks: constructor public XXX is not whitelisted

I was using Azure Databricks and trying to run some example python code from this page.

But I get this exception:

py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.classification.LogisticRegression(java.lang.String) is not whitelisted.

Upvotes: 16

Views: 33693

Answers (1)

Dustin V
Dustin V

Reputation: 336

This error shows up with some library methods when using High Concurrency cluster with credential pass through enabled. If that is your scenario a work around that may be an option is to use a different cluster mode.

py4j.security.Py4JSecurityException: ... is not whitelisted This exception is thrown when you have accessed a method that Azure Databricks has not explicitly marked as safe for Azure Data Lake Storage credential passthrough clusters. In most cases, this means that the method could allow a user on a Azure Data Lake Storage credential passthrough cluster to access another user’s credentials.

Reference: https://learn.microsoft.com/en-us/azure/databricks/data-governance/credential-passthrough/adls-passthrough

Upvotes: 18

Related Questions