CODEWITHSUNDEEP
eclipsescalamavenapache-sparkapache-spark-sql
BludShot
BludShot

Reputation: 301

Using HiveContext in Maven project

I have built Spark-1.2.1 using Maven to enable Hive support using the following command : mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -DskipTests clean package

which resulted in some class files generated in /spark-1.2.1/core/target/scala-2.10/classes folder

Now how do I use this newly built Spark in my Eclipse + Maven project? I want to use the Spark-SQL's HiveContext class in Eclipse.

Upvotes: 3

Views: 10693

Answers (1)

vikas
vikas

Reputation: 1558

When you are going to use SparkSQL in java program, you can simply add the appropriate dependencies to your maven project and the required classed would be available to you. Building spark with hive support enables you to start spark daemons with hive support. It generates an assembly jar which needed to copied to all spark worker nodes. For more details please see here

Maven dependencies for getting HiveContext working:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>1.2.1</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.10</artifactId>
    <version>1.2.1</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-hive_2.10</artifactId>
    <version>1.2.1</version>
</dependency>

Upvotes: 18

Related Questions