CODEWITHSUNDEEP
apache-sparkhive
David H
David H

Reputation: 1356

Hive on spark, partition pruning, better undertanding

I have a spark 1.6.2 code using SQL/HQL language. I really tried to understand if my job is doing partition pruning or not. Data is partitioned by date (cdate field) the explain plan is : 

== Physical Plan ==
Project [coalesce(cdate#74,cdate#38) AS cdate#29,coalesce(account_key#75,account_key#34) AS account_key#30,coalesce(product#76,product#35) AS product#31,(coalesce(amount#77,0.0) + coalesce(amount#36,0.0)) AS amount#32,(coalesce(volume#78L,0) + cast(coalesce(volume#37,0) as bigint)) AS volume#33L]
+- SortMergeOuterJoin [account_key#34,cdate#38,product#35], [account_key#75,cdate#74,product#76], FullOuter, None
   :- Sort [account_key#34 ASC,cdate#38 ASC,product#35 ASC], false, 0
   :  +- TungstenExchange hashpartitioning(account_key#34,cdate#38,product#35,200), None
   :     +- Project [volume#37,product#35,cdate#38,account_key#34,amount#36]
   :        +- BroadcastHashJoin [cdate#38], [cdate#24], BuildLeft
   :           :- Scan ParquetRelation[account_key#34,product#35,amount#36,volume#37,cdate#38] InputPaths: hdfs://hdp1.voicelab.local:8020/apps/hive/warehouse/my.db/daily_profiles
   :           +- TungstenAggregate(key=[cdate#24], functions=[], output=[cdate#24])
   :              +- TungstenExchange hashpartitioning(cdate#24,200), None
   :                 +- TungstenAggregate(key=[cdate#24], functions=[], output=[cdate#24])
   :                    +- Project [cdate#24]
   :                       +- TungstenAggregate(key=[cdate#20,accountKey#21,product#22], functions=[], output=[cdate#24])
   :                          +- TungstenExchange hashpartitioning(cdate#20,accountKey#21,product#22,200), None
   :                             +- TungstenAggregate(key=[cdate#20,accountKey#21,product#22], functions=[], output=[cdate#20,accountKey#21,product#22])
   :                                +- Project [cdate#20,accountKey#21,product#22]
   :                                   +- Scan ExistingRDD[cdate#20,accountKey#21,product#22,amount#23]
   +- Sort [account_key#75 ASC,cdate#74 ASC,product#76 ASC], false, 0
      +- TungstenExchange hashpartitioning(account_key#75,cdate#74,product#76,200), None
         +- TungstenAggregate(key=[cdate#20,accountKey#21,product#22], functions=[(sum(amount#23),mode=Final,isDistinct=false),(count(1),mode=Final,isDistinct=false)], output=[cdate#74,account_key#75,product#76,amount#77,volume#78L])
            +- TungstenExchange hashpartitioning(cdate#20,accountKey#21,product#22,200), None
               +- TungstenAggregate(key=[cdate#20,accountKey#21,product#22], functions=[(sum(amount#23),mode=Partial,isDistinct=false),(count(1),mode=Partial,isDistinct=false)], output=[cdate#20,accountKey#21,product#22,sum#54,count#55L])
                  +- Scan ExistingRDD[cdate#20,accountKey#21,product#22,amount#23]

How can I figure out if my job is using the metastore in order to do partition pruning.

Can you elaborate about Scan ParquetRelation? how can I know that the scan using partition pruning/discovery ? what is the meaning for the field#SOME_NUMBER i.e account_key#34

The use case is aggregating data per date,account,product

Upvotes: 1

Views: 154

Answers (1)

Nitin Kumar
Nitin Kumar

Reputation: 249

Look for PartitionFilters: [... ] in the Physical plan. If the array has a non empty value, it's using otherwise no. I couldn't find in your plan, unless I missed it or could not find it.

Upvotes: 1

Related Questions