CODEWITHSUNDEEP
apache-sparkpyspark
naveed mohad abdul
naveed mohad abdul

Reputation: 113

Why does pyspark fail with "Unable to locate hive jars to connect to metastore. Please set spark.sql.hive.metastore.jars."?

I am using a standalone cluster of apache spark version 2.0.0 with two nodes and i have not installed hive.I am getting the following error on creating a dataframe.

from pyspark import SparkContext
from pyspark import SQLContext
sqlContext = SQLContext(sc)
l = [('Alice', 1)]
sqlContext.createDataFrame(l).collect()
---------------------------------------------------------------------------
IllegalArgumentException                  Traceback (most recent call last)
<ipython-input-9-63bc4f21f23e> in <module>()
----> 1 sqlContext.createDataFrame(l).collect()

/home/mok/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/context.pyc in createDataFrame(self, data, schema, samplingRatio)
    297         Py4JJavaError: ...
    298         """
--> 299         return self.sparkSession.createDataFrame(data, schema, samplingRatio)
    300 
    301     @since(1.3)

/home/mok/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/session.pyc in createDataFrame(self, data, schema, samplingRatio)
    522             rdd, schema = self._createFromLocal(map(prepare, data), schema)
    523         jrdd = self._jvm.SerDeUtil.toJavaArray(rdd._to_java_object_rdd())
--> 524         jdf = self._jsparkSession.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
    525         df = DataFrame(jdf, self._wrapped)
    526         df._schema = schema

/home/mok/spark-2.0.0-bin-hadoop2.7/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
    931         answer = self.gateway_client.send_command(command)
    932         return_value = get_return_value(
--> 933             answer, self.gateway_client, self.target_id, self.name)
    934 
    935         for temp_arg in temp_args:

/home/mok/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/utils.pyc in deco(*a, **kw)
     77                 raise QueryExecutionException(s.split(': ', 1)[1], stackTrace)
     78             if s.startswith('java.lang.IllegalArgumentException: '):
---> 79                 raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
     80             raise
     81     return deco

IllegalArgumentException: u'Unable to locate hive jars to connect to metastore. Please set spark.sql.hive.metastore.jars.'

So should i install Hive or edit the configurations.

Upvotes: 11

Views: 14564

Answers (5)

Abhay S
Abhay S

Reputation: 21

If you have multiple jdks installed, you can find the java homes as below

/usr/libexec/java_home -V
Matching Java Virtual Machines (3):
    13.0.2, x86_64: "OpenJDK 13.0.2"    /Library/Java/JavaVirtualMachines/adoptopenjdk-13.0.2.jdk/Contents/Home
    11.0.6, x86_64: "AdoptOpenJDK 11"   /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
    1.8.0_252, x86_64:  "AdoptOpenJDK 8"    /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home

Now set the JAVA_HOME for 1.8 use

export JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home

Upvotes: 2

ashwin kumar
ashwin kumar

Reputation: 191

IllegalArgumentException: u'Unable to locate hive jars to connect to metastore. Please set spark.sql.hive.metastore.jars.'

I had the same issue and fixed it by using Java 8. Make sure you install JDK 8 and set the environment variables accordingly.

Do not use Java 11 with Spark / pyspark 2.4.

Upvotes: 19

jeremy_rutman
jeremy_rutman

Reputation: 5720

If you have several java versions you'll have to figure out which spark is using (I did this using trial and error , starting with 

JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64"

and ending with 

JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"

Upvotes: 15

Tomer Ben David
Tomer Ben David

Reputation: 8886

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home

Did the trick.

Upvotes: 3

Pranay Aryal
Pranay Aryal

Reputation: 5396

Please ensure that your JAVA_HOME environment variable is set. For Mac OS I did, echo export JAVA_HOME=/Library/Java/Home >> ~/.bash_profile and then source ~/.bash_profile or open ~/.bash_profile in type the above.

Upvotes: 0

Related Questions