CODEWITHSUNDEEP
hadoopmapreducehive
Lazer
Lazer

Reputation: 94800

How does Hive decide when to use map reduce and when not to?

As a simple example,

select * from tablename;

DOES NOT kick in map reduce, while

select count(*) from tablename;

DOES. What is the general principle used to decide when to use map reduce (by hive)?

Upvotes: 10

Views: 12485

Answers (4)

Pardeep Sharma
Pardeep Sharma

Reputation: 582

Whenever we fire a query like select * from tablename, Hive reads the data file and fetches the entire data without doing any aggregation(min/max/count etc.). It'll call a FetchTask rather than a mapreduce task.

This is also an optimization technique in Hive. hive.fetch.task.conversion property can (i.e. FETCH task) minimize latency of map-reduce overhead.

This is like we are reading a hadoop file : hadoop fs -cat filename

But if we use select colNames from tablename, it requires a map-reduce job as it needs to extract the 'column' from each row by parsing it from the file it loads.

Upvotes: 6

user6260103
user6260103

Reputation: 1

It is an optimisation technique, hive.fetch.task.conversion property can (FETCH) task minimize latency of mapreduce overhead.

When doing SELECT, LIMIT, FETCH queries this property skips mapreduce and uses the FETCH task.

This property can have 3 values - none, minimal (the default) and more.

Upvotes: -1

wlk
wlk

Reputation: 5785

select * from tablename;

Just reads raw data from files in HDFS, so it is much faster without MapReduce.

Upvotes: 1

Donald Miner
Donald Miner

Reputation: 39893

In general, any sort of aggregation, such as min/max/count is going to require a MapReduce job. This isn't going to explain everything for you, probably.

Hive, in the style of many RDBMS, has an EXPLAIN keyword that will outline how your Hive query gets translated into MapReduce jobs. Try running explain on both your example queries and see what it is trying to do behind the scenes.

Upvotes: 10

Related Questions