CODEWITHSUNDEEP
hive
Mayank Jaiswal
Mayank Jaiswal

Reputation: 13096

What are the advantages of setting "hive.exec.parallel" to false in Hive ?

I came to know that when hive.exec.parallel is set to true in hive i.e 

set hive.exec.parallel=true;

then independent tasks in a query can run in parallel.

Thanks to Qubole for this: hive parallel execution

Are there any advantages of setting this parameter to false? I'll iterate myself here: Obviously, whenever possible, you would like to run things in parallel and have more throughput. Why would someone set this parameter to false - are there any disadvantages too?

Upvotes: 23

Views: 27201

Answers (3)

Bector
Bector

Reputation: 1334

Mayank, This property also has benefits with some star condition. I mean to say that Hive has a feature of database locking while multiple queries running on that database.

For example -
You have a complex query with multiple stages running on one database where Parallel property can increase your efficiency but It will also create "LOCK" on DATABASE which may stop other processes which are running on same database for the time of it's own execution.

I have recently faced this issue and resolved by making this property "FALSE".
I hope this answer may help you to understand in what scenario we have to make it false.

Upvotes: 3

Guy Needham
Guy Needham

Reputation: 390

In my experience, the only disadvantage is resource use. If you have limited resources available, it could be better overall to have queries running serially. When queries run in parallel, one query can manage several jobs at the same time, which could starve the cluster of resources. If you don't need the speed and have a cluster with a lot of workload, it might be better overall to let things run serially.

Upvotes: 4

Bennie Schut
Bennie Schut

Reputation: 185

It's simply a parameter because when it got introduced it wasn't clear how stable it would be and so you should be able to turn it off. Once enough people tried it and found it stable the default switched to true: https://issues.apache.org/jira/browse/HIVE-1033

There is no realistic disadvantage at this time.

Upvotes: 7

Related Questions