CODEWITHSUNDEEP
apache-sparkpysparkdatabricks
Edward Ashak
Edward Ashak

Reputation: 2441

Can I use a python function in DataFrame.selectExpr

I've got a DataFrame that I'm executing a selectExpr on

selectExpr("src.ID", "dst.ID", "SQRT(POW((src.X - dst.X),2) + POW((src.Y - dst.Y),2)) as dist") \

is there any way I could do the path on the dist column in a Python function ?

Upvotes: 0

Views: 1549

Answers (1)

Haroun Mohammedi
Haroun Mohammedi

Reputation: 2434

I'm comming from scala but there's obviously a similar way in python. the udf function from the sql.fucnctions package allow developers to create there's own UserDefinedFunction and using them inside the spark Dataframe operation and SQL queries. here's the code that can perform what you've asked for

val constumeFunction = udf((x: Double, y: Double, z: Double, t: Double) => pow(x - y, 2) + pow(z - t, 2))

then you can call this UserDefinedFunction inside the select method like this :

selectExpr($"src.ID", $"dst.ID",costumeFunction(src.X, dst.X, src.Y, dst.Y) as dist")

sorry for answering with Scala code but I'm sure that there's a very similar way in python

Upvotes: 1

Related Questions