CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sql
Chandan392
Chandan392

Reputation: 57

what is the pyspark package equivalent to org.apache.spark.sql.execution?

I have the below code, which is in PySpark,

df1 = spark.range(2, 10000000, 2)
df2 = spark.range(2, 10000000, 4)
step1 = df1.repartition(5)
step12 = df2.repartition(6)
step2 = step1.selectExpr("id * 5 as id")
step3 = step2.join(step12, ["id"])
step4 = step3.selectExpr("sum(id)")
step4.collect()

I want to use step4.queryExecution.debug.codegen, which is present in Scala package. Can anyone point me to the current package.

Upvotes: 2

Views: 211

Answers (1)

10465355
10465355

Reputation: 4631

This part of the API is not exposed in PySpark, but it can be reached through the internal JVM object:

step4._jdf.queryExecution().debug().codegen()

Upvotes: 3

Related Questions