CODEWITHSUNDEEP
hivehiveqlexplain
Markiza
Markiza

Reputation: 444

Hive explain plan where to see full table scan?

How can I see from hive EXPLAIN is there a full table scan?

For example, is there a full scan? The table size is 993 rows.

The query is

explain select latitude,longitude FROM CRIMES WHERE geohash='dp3twhjuyutr'

I have secondary index on geohash column.

STAGE PLANS:
      Stage: Stage-1
        Map Reduce
          Map Operator Tree:
              TableScan
                alias: crimes
                filterExpr: (geohash = 'dp3twhjuyutr') (type: boolean)
                Statistics: Num rows: 993 Data size: 265582 Basic stats: COMPLETE Column stats: NONE
                Filter Operator
                  predicate: (geohash = 'dp3twhjuyutr') (type: boolean)
                  Statistics: Num rows: 496 Data size: 132657 Basic stats: COMPLETE Column stats: NONE
                  Select Operator
                    expressions: latitude (type: double), longitude (type: double)
                    outputColumnNames: _col0, _col1
                    Statistics: Num rows: 496 Data size: 132657 Basic stats: COMPLETE Column stats: NONE
                    File Output Operator
                      compressed: false
                      Statistics: Num rows: 496 Data size: 132657 Basic stats: COMPLETE Column stats: NONE
                      table:
                          input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                          output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                          serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

  Stage: Stage-0
    Fetch Operator
      limit: -1
      Processor Tree:
        ListSink

Upvotes: 3

Views: 3253

Answers (1)

leftjoin
leftjoin

Reputation: 38335

  1. Absence of partition predicate in the plan means full scan. Of course this is not about predicate push-down in ORC.
  2. Check Data size and Num rows in each operator.
  3. EXPLAIN DEPENDENCY command will show all input_partitions collection and you can check what exactly will be scanned.

Upvotes: 3

Related Questions