CODEWITHSUNDEEP
apache-sparkpysparkdatabricks
jjc
jjc

Reputation: 33

java.lang.SecurityException: Your administrator has forbidden Scala UDFs from being run on this cluster

I am working on a standard concurrency databricks cluster with the following configurations: 8.1 (includes Apache Spark 3.1.1, Scala 2.12) as well as this set in config spark.databricks.repl.allowedLanguages python,sql,scala

When I try to fit my NLP pipeline using this code nlp_model = nlp_pipeline.fit(data) I am getting this error: java.lang.SecurityException: Your administrator has forbidden Scala UDFs from being run on this cluster

Does anyone have any ideas how to get around this please? I think this error should not appear on a standard concurrency cluster (and is why i set this one up). Our IT team are not able to help me sadly :(

and this worked last week!

Thanks in advance James

Upvotes: 1

Views: 1321

Answers (1)

dyno
dyno

Reputation: 21

I believe you may be using a runtime of Databricks that does not support the SCALA cmdlets. It could be likely that you are using a cluster with AD pass through enabled. Can you please try the same cmdlets without AD Pass through and see if that works?

Upvotes: 2

Related Questions