CODEWITHSUNDEEP
hadoopmapreducehive
wandermonk
wandermonk

Reputation: 7356

How to set the resources for different hive stages from the EXPLAIN?

Is there any way to provide resources basing on the stages from the EXPLAIN query.

Upvotes: 1

Views: 2382

Answers (1)

Samson Scharfrichter
Samson Scharfrichter

Reputation: 9067

AFAIK the only things you can set are:

  • number of R / exact count set mapred.reduce.tasks=** (as long as it makes sense)
  • number of R / max count set hive.exec.reducers.max=** + hint for computing the actual number set hive.exec.reducers.bytes.per.reducer=*****
  • resources per M container (when using MR) / set mapreduce.map.memory.mb=** + set mapreduce.map.java.opts=-XMX** <~80% of total for heap>
  • resources per R container (when using MR) / set mapreduce.reduce.memory.mb=** + set mapreduce.reduce.java.opts=-XMX** <~80% of total for heap>
  • resources per container (when using TEZ) / set hive.tez.container.size=** + set hive.tez.java.opts=-XMX** <~80% of total for heap>

Note that if your query shows 3 Map steps and 2 Reduce steps, the settings will apply to all steps...

Reference : Hive documentation and that post for instance

Upvotes: 3

Related Questions