CODEWITHSUNDEEP
javapythonapache-sparkpysparkspark-hive
masta-g3
masta-g3

Reputation: 1222

HiveContext createDataFrame not working on pySpark (jupyter)

I am doing an analysis on pySpark using the Jupyter notebooks. My code originally build dataframes using sqlContext = SQLContext(sc), but now I've switched to HiveContext since I will be using window functions.

My problem is that now I'm getting a Java error when trying to create the dataframe:

## Create new SQL Context.
from pyspark.sql import SQLContext
from pyspark.sql import DataFrame
from pyspark.sql import Window
from pyspark.sql.types import *
import pyspark.sql.functions as func

sqlContext = HiveContext(sc)

After this I read my data into an RDD, and create the schema for my DF.

## After loading the data we define the schema.
fields = [StructField(field_name, StringType(), True) for field_name in data_header]
schema = StructType(fields)

Now, when I try to build the DF this is the error I get:

## Build the DF.
data_df = sqlContext.createDataFrame(data_tmp, schema)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
...
/home/scala/spark-1.6.1/python/pyspark/sql/context.pyc in _get_hive_ctx(self)
    690 
    691     def _get_hive_ctx(self):
--> 692         return self._jvm.HiveContext(self._jsc.sc())
    693 
    694     def refreshTable(self, tableName):

TypeError: 'JavaPackage' object is not callable

I have been googling it without luck so far. Any advice is greatly appreciated.

Upvotes: 3

Views: 1846

Answers (1)

zero323
zero323

Reputation: 330063

HiveContext requires binaries build with Hive support. It means you have to enable Hive profile. Since you use sbt assembly you need at least:

sbt -Phive assembly

The same is required when building with Maven, for example:

mvn -Phive -DskipTests clean package

Upvotes: 1

Related Questions