CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sqluser-defined-functions
Tarun Kumar Yellapu
Tarun Kumar Yellapu

Reputation: 562

Does spark GROUPED_MAP udf on a data frame run parallelly?

I'm trying to apply a PandasUDFType.GROUPED_MAP function which takes a data frame as input and yields a data frame as an output. When I do sdf.groupby(key).apply(pandas_udf) does it apply the function parallelly to multiple groups based on available resources or sequentially one group after the other? I haven't changed any default settings of spark. What other alternatives can I employ if I want to execute udf on groups parallelly.

Upvotes: 0

Views: 499

Answers (1)

uxke
uxke

Reputation: 456

Yes UDFs are executed parallel but the execution is not as optimized as spark native functions.

More info here: Spark functions vs UDF performance?

Upvotes: 1

Related Questions