CODEWITHSUNDEEP
hadoophivebigdata
Lisha Priyadharshini
Lisha Priyadharshini

Reputation: 25

how indexing works internally in hive?

An Index is nothing but a pointer on a particular column of a table. Creating an index means creating a pointer on a particular column of a table. If a column is indexed in a table, and how the data of that particular column is pointed,when that specific column is queried?

Upvotes: 1

Views: 2570

Answers (1)

hlagos
hlagos

Reputation: 7947

From the documentation

The goal of Hive indexing is to improve the speed of query lookup on certain columns of a table. Without an index, queries with predicates like 'WHERE tab1.col1 = 10' load the entire table or partition and process all the rows. But if an index exists for col1, then only a portion of the file needs to be loaded and processed. The improvement in query speed that an index can provide comes at the cost of additional processing to create the index and disk space to store the index.

Behind the scene, Hive creates essentially a Map with the values of the column that it is indexing and the offset + files where the data is located in the HDFS, in that way, Hive does not need to scan all the data to search for a certain value. Here is a good article explaining basic concepts

https://acadgild.com/blog/indexing-in-hive/

Upvotes: 2

Related Questions