CODEWITHSUNDEEP
pythonhiveconfigurationkerberospyhive
cfrapper
cfrapper

Reputation: 55

Connection to Hive using python and Kerberos

I'm trying to connect to hive using Python. I installed all of the dependencies required (sasl, thrift_sasl, etc..)

Here is how I try to connect:

configuration = {"hive.server2.authentication.kerberos.principal" : "hive/_HOST@REALM_HOST", "hive.server2.authentication.kerberos.keytab" : "/etc/security/keytabs/hive.service.keytab"}

connection = hive.Connection(configuration = configuration, host="host", port=port, auth="KERBEROS", kerberos_service_name = "hiveserver2")

But I get this error:

Minor code may provide more information (Cannot find KDC for realm "REALM_DOMAIN")

Whay I'm missing? Does someone has an example of an pyHive connection using kerberos?

Thank you for your help.

Upvotes: 1

Views: 6946

Answers (2)

cfrapper
cfrapper

Reputation: 55

Thank you @Kishore. Actually in PySpark, the code looks like this : 

import pyspark
from pyspark import SparkContext
from pyspark.sql import Row
from pyspark import SparkConf
from pyspark.sql import HiveContext
from pyspark.sql import functions as F
import pyspark.sql.types as T

def connection(self):
    conf = pyspark.SparkConf()
    conf.setMaster('yarn-client')
    sc = pyspark.SparkContext(conf=conf)

    self.cursor = HiveContext(sc)

    self.cursor.setConf("hive.exec.dynamic.partition", "true")
    self.cursor.setConf("hive.exec.dynamic.partition.mode", "nonstrict")
    self.cursor.setConf("hive.warehouse.subdir.inherit.perms", "true")
    self.cursor.setConf('spark.scheduler.mode', 'FAIR')

and you can request using :

rows = self.cursor.sql("SELECT someone FROM something")
for row in rows.collect():
    print row

I'm actually running the code via the command :

spark-submit --master yarn MyProgram.py

I guess you could using basically run the python with pyspark installed like : 

python MyProgram.py

but I didn't tried so I won't assure that it's working

Upvotes: 1

Kishore
Kishore

Reputation: 5881

I don't know in pyspark, but I am using below scala code and it is working since last one year. If you can change this code in python. Replace the value of properties based on your kerberos.

System.setProperty("hive.metastore.uris", "add hive.metastore.uris url");
System.setProperty("hive.metastore.sasl.enabled", "true")
System.setProperty("hive.metastore.kerberos.keytab.file", "add keytab")
System.setProperty("hive.security.authorization.enabled", "false")
System.setProperty("hive.metastore.kerberos.principal", "replace hive.metastore.kerberos.principal value")
System.setProperty("hive.metastore.execute.setugi", "true")
val hiveContext = new HiveContext(sparkContext)

Upvotes: 0

Related Questions