CODEWITHSUNDEEP
apache-sparkapache-spark-sql
San
San

Reputation: 83

Spark - How can get the Logical / Physical Query execution using - Thirft - Hive Interactor

Spark - How can get the Logical / Physical Query execution using the following

  1. Via Thrift
  2. Via SparkInteractor

Upvotes: 7

Views: 21658

Answers (2)

Joy Jedidja Ndjama
Joy Jedidja Ndjama

Reputation: 582

If you are using Spark 1, You can get the explain on a query this way:

sqlContext.sql("your SQL query").explain(true)

If you are using Spark 2, it's the same:

spark.sql("your SQL query").explain(true)

The same logic is available on a dataframe:

yourDF.explain(true) or yourDF.someOperation.explain(true)

Where someOperation could be maybe a select on a join or something else.

Upvotes: 3

Srinivasarao Daruna
Srinivasarao Daruna

Reputation: 3374

You can use explain statement with query as below in beeline via thrift. 

EXPLAIN EXTENDED select * from sr23 join sr12 [<join condidtion>]

What do you mean be spark interceptor.? is it spark-sql shell.? if it is, then you can use above query.

If you meant spark-shell, then you need to call explain() function on dataframes.

eg:

val df1 = sqlContext.sql(" < your sql query > ");

df1.explain;

this will give both physical and logical plans. You can also see them from spark web UI in SQL tab.

Upvotes: 5

Related Questions