CODEWITHSUNDEEP
pythonapache-sparkpysparkuser-defined-functions
abc
abc

Reputation: 177

TypeError: udf() missing 1 required positional argument: 'f'

I couldn't find any solution or question to my problem.

If I try to define a Spark-UDF Function (pyspark) e.g.:

@udf(returnType=IntegerType())
def _return_an_integer(y):

    return y + random.randint(1, 99)

df = df.withColumn('randomInteger', _return_an_integer(col('1')))

I get the error message:

TypeError: udf() missing 1 required positional argument: 'f'

Has anybody a solution for this?

Thanks in advance!

EDIT: Without annotation:

Py4JError: An error occurred while calling None.org.apache.spark.sql.execution.python.UserDefinedPythonFunction. Trace:
py4j.Py4JException: Constructor org.apache.spark.sql.execution.python.UserDefinedPythonFunction([class java.lang.String, class org.apache.spark.api.python.PythonFunction, class org.apache.spark.sql.types.IntegerType$]) does not exist
    at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:179)
    at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:196)
    at py4j.Gateway.invoke(Gateway.java:237)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)

Upvotes: 0

Views: 1229

Answers (1)

abc
abc

Reputation: 177

After trying lot of things, the problem was that my pyspark version didn't match the spark version.

Upvotes: 1

Related Questions