CODEWITHSUNDEEP
hortonworks-data-platformambari
Knows Not Much
Knows Not Much

Reputation: 31546

How to install Apache Spark on HortonWorks HDP 2.2 (built using Ambari)

I successfully built a 5 node cluster of HortonWorks HDP 2.2 using Ambari.

However I don't see Apache Spark in the installed services list.

I did some research and found that Ambari does not install certain components like hue etc. ( Spark was not in that list, but I guess its not installed).

How do I do a manual install of Apache spark on my 5 node HDP 2.2?

Or should I delete my cluster and perform a fresh install without using Ambari?

Upvotes: 3

Views: 7296

Answers (3)

abdielou
abdielou

Reputation: 306

You could build your own Ambari Stack for Spark. I recently did just that, but I cannot share that code :(

What I can do is share a tutorial I did on how to do any stack for Ambari, including Spark. There are many interesting issues with Spark that need to be addressed and are not covered through the tutorial. Anyways hope it helps. http://bit.ly/1HDBgS6

There is also a guide from the Ambari people here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=38571133.

Upvotes: 1

Ali Rezvanian
Ali Rezvanian

Reputation: 66

1) Ambari 1.7x does not install Accumulo, Hue, Ranger, or Solr services for the HDP 2.2 Stack. For Installing Accumulo, Hue, Knox, Ranger, and Solr services, install HDP Manually.

2) Apache Spark 1.2.0 on YARN with HDP 2.2 : here .

3) Spark and Hadoop: Working Together :

Standalone deployment: With the standalone deployment one can statically allocate resources on all or a subset of machines in a Hadoop cluster and run Spark side by side with Hadoop MR. The user can then run arbitrary Spark jobs on her HDFS data. Its simplicity makes this the deployment of choice for many Hadoop 1.x users.

Hadoop Yarn deployment: Hadoop users who have already deployed or are planning to deploy Hadoop Yarn can simply run Spark on YARN without any pre-installation or administrative access required. This allows users to easily integrate Spark in their Hadoop stack and take advantage of the full power of Spark, as well as of other components running on top of Spark.

Spark In MapReduce : For the Hadoop users that are not running YARN yet, another option, in addition to the standalone deployment, is to use SIMR to launch Spark jobs inside MapReduce. With SIMR, users can start experimenting with Spark and use its shell within a couple of minutes after downloading it! This tremendously lowers the barrier of deployment, and lets virtually everyone play with Spark.

Upvotes: 0

philshem
philshem

Reputation: 25331

Hortonworks support for Spark is arriving but not fully complete (details and blog).

Instructions for how to integrate Spark with HDP can be found here.

Upvotes: 1

Related Questions