CODEWITHSUNDEEP
pysparkapache-spark-sql
M4rk
M4rk

Reputation: 2272

Disable some functions in PySpark sql

Is there a way to prevent users to use certain functions in PySpark SQL?

e.g. Let's say I want to avoid users using function log or rand, how could I disable it?

Upvotes: 0

Views: 135

Answers (1)

werner
werner

Reputation: 14895

You can register an udf with the same name as the function that you want to disable and throw an exception in the udf:

def log(s):
    raise Exception("log does not work anymore")
spark.udf.register("log", log)

spark.sql("select *, log(value) from table").show()

Result:

PythonException: 
  An exception was thrown from the Python worker. Please see the stack trace below.
Traceback (most recent call last):
[...]
Exception: log does not work anymore

Upvotes: 2

Related Questions