CODEWITHSUNDEEP
apache-sparkpyspark
eugene
eugene

Reputation: 41665

Spark, how to print the query?

I'm using pyspark 

    df = self.sqlContext.read.option(
        "es.resource", indexes
    ).format("org.elasticsearch.spark.sql").load()


    df = df.filter(
        df.data.timestamp >= self.period_start
    )

I'd like to see sql query version of df if possible.

something like print(df.query) to see something like select * from my-indexes where data.timestamp > self.period_start

Upvotes: 2

Views: 7603

Answers (1)

pissall
pissall

Reputation: 7399

You can check out this piece of documentation for pyspark.sql.DataFrame.explain . explain prints the (logical and physical) plan to the console for debugging purpose.

Upvotes: 3

Related Questions