CODEWITHSUNDEEP
pythonapache-sparkpysparkdatabricks
Yoren Van Couwenberghe
Yoren Van Couwenberghe

Reputation: 53

Pyspark error with UDF: py4j.Py4JException: Method __getnewargs__([]) does not exist error

I am trying to solve the following error (I am using the databricks platform and spark 2.0)

tweets_cleaned.createOrReplaceTempView("tweets_cleanedSQL")
def Occ(keyword):
  occurences = spark.sql("SELECT * \
                                FROM tweets_cleanedSQL \
                                WHERE LOWER(text) LIKE '%" + keyword + "%' \
                            ")
  return occurences.count()


occurences_udf = udf(Occ)

If I run this code, I receive the following error:

py4j.Py4JException: Method getnewargs([]) does not exist ==> error only occurs when trying to define the udf.

Upvotes: 4

Views: 5436

Answers (1)

Assaf Mendelson
Assaf Mendelson

Reputation: 12991

the UDF function turns a regular function to a function that is applied on any element of the input column. You cannot have this function call spark functions (in this case you are calling spark.sql which would in turn need to create workers etc. which is not supported.

Upvotes: 8

Related Questions