CODEWITHSUNDEEP
hadoophive
Minwoo
Minwoo

Reputation: 243

always Hive Job running in-process local Hadoop

When I set this property in hive-site.xml

<property>
  <name>hive.exec.mode.local.auto</name>
  <value>false</value>
</property>

Hive always runs the hadoop job locally.

Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 55
Job running in-process (local Hadoop)

Why does this happen?

Upvotes: 1

Views: 3022

Answers (1)

Vinkal
Vinkal

Reputation: 2984

As mentioned in HIVE-2585,Going forward Hive will assume that the metastore is operating in local mode if the configuration property hive.metastore.uris is unset, and will assume remote mode otherwise.

Ensure following property is set in Hive-site.xml:

<property>
    <name>hive.metastore.uris</name>
    <value><URIs of metastore server>:9083</value>
</property>
<property>
    <name> hive.metastore.local</name>
    <value>false</value> 
</property>

The hive.metastore.local property is no longer supported as of Hive 0.10; setting hive.metastore.uris is sufficient to indicate that you are using a remote metastore.

EDIT:

Starting with release 0.7, Hive also supports a mode to run map-reduce jobs in local-mode automatically. The relevant options are hive.exec.mode.local.auto, hive.exec.mode.local.auto.inputbytes.max, and hive.exec.mode.local.auto.tasks.max:

hive> SET hive.exec.mode.local.auto=false;

Note that this feature is disabled by default. If enabled, Hive analyzes the size of each map-reduce job in a query and may run it locally if the following thresholds are satisfied:

1. The total input size of the job is lower than: hive.exec.mode.local.auto.inputbytes.max (128MB by default)

2. The total number of map-tasks is less than: hive.exec.mode.local.auto.tasks.max (4 by default)

3. The total number of reduce tasks required is 1 or 0.

So for queries over small data sets, or for queries with multiple map-reduce jobs where the input to subsequent jobs is substantially smaller (because of reduction/filtering in the prior job), jobs may be run locally.

Reference: Hive Getting started

Upvotes: 2

Related Questions