CODEWITHSUNDEEP
pyspark
xiaodai
xiaodai

Reputation: 16014

Is it possible to construct a PySpark statement from strings?

Consider

from pyspark.sql import functions as F

statement = F.when(F.col("col1") < 0.5, F.lit(0.5))

str_statement = str(statement)

yields

"Column<'CASE WHEN (col1 < 0.5) THEN 0.5 END'>"

Is it possible to take a string like this and without writing a custom parser recreate the statement?

Upvotes: 1

Views: 60

Answers (1)

werner
werner

Reputation: 14845

The sql part of the string can be used as parameter for expr to recreate the original column object.

str_statement = "Column<'CASE WHEN (col1 < 0.5) THEN 0.5 END'>"

import re
sql_expr = re.search("Column<'(.*)'>", str_statement).group(1)

from pyspark.sql import functions as F
statement = F.expr(sql_expr)

Upvotes: 1

Related Questions