Reputation: 57
I have the below code, which is in PySpark,
df1 = spark.range(2, 10000000, 2)
df2 = spark.range(2, 10000000, 4)
step1 = df1.repartition(5)
step12 = df2.repartition(6)
step2 = step1.selectExpr("id * 5 as id")
step3 = step2.join(step12, ["id"])
step4 = step3.selectExpr("sum(id)")
step4.collect()
I want to use step4.queryExecution.debug.codegen
, which is present in Scala package. Can anyone point me to the current package.
Upvotes: 2
Views: 211
Reputation: 4631
This part of the API is not exposed in PySpark, but it can be reached through the internal JVM object:
step4._jdf.queryExecution().debug().codegen()
Upvotes: 3