CODEWITHSUNDEEP
hivepysparkapache-spark-sql
Krishna
Krishna

Reputation: 1209

How can I connect to hive using pyspark?

I'm trying to create a table in HIVE. But it is creating a folder like testdb.db inside spark-warehouse folder. How can I directly store in HIVE as we store to MySQL/MongoDB databases.

conf = SparkConf().setAppName("data_import")
sc = SparkContext(conf = conf)

sqlContext = HiveContext(sc)

sqlContext.setConf("spark.sql.shuffle.partitions", "2")
sqlContext.sql("CREATE DATABASE testdb")
sqlContext.sql("use testdb")
sqlContext.sql("create table daily_revenue(order_date string, daily_revenue float)")

Upvotes: 1

Views: 1925

Answers (2)

Bala
Bala

Reputation: 11234

sqlContext.sql("create database if not exists demo")

>>> sqlContext.sql("show tables in demo").show()
+---------+-----------+
|tableName|isTemporary|
+---------+-----------+
+---------+-----------+

sqlContext.sql("create table demo.dummy (id int, name string)")

>>> sqlContext.sql("show tables in demo").show()
+---------+-----------+
|tableName|isTemporary|
+---------+-----------+
|    dummy|      false|
+---------+-----------+

>>> sqlContext.sql("desc demo.dummy").show()
+--------+---------+-------+
|col_name|data_type|comment|
+--------+---------+-------+
|      id|      int|   null|
|    name|   string|   null|
+--------+---------+-------+

Upvotes: -1

Neeraj Bhadani
Neeraj Bhadani

Reputation: 3100

When you creates a table in HIVE then what happens behind the scene is, it stores the metadata in some relational database depending on which is configured for your environment and actual data will be stored on HDFS warehouse directory if that is managed table.

Similarly when you try to create the table from Spark in HIVE then what it will do is, first it will create the folder .db and inside this folder it will create another folder with table name, which inturn store the data on HDFS.

So in your case, you should have <warehouse_dir>/testdb.db/table folder. and if you load any data to this table, that will be present inside the table folder.

Hope it helps.

Regards,

Neeraj

Upvotes: 2

Related Questions