Installing spark on hadoop

Question

I installed hadoop 2.7 on my mac. Then i want to install spark on it. But there is no any document for this.can anybody explain step by step how to install spark on hadoop?

Answer 1

Here are the steps I took to install Apache Spark to a Linux Centos system with hadoop:

• Install a default Java system (ex: sudo yum install java-11-openjdk)
• Download latest release of Apache Spark from spark.apache.org
• Extract the Spark tarball (tar xvf spark-2.4.5-bin-hadoop2.7.tgz)
• Move Spark folder created after extraction to the /opt/ directory (sudo mv spark-2.4.5-bin-hadoop2.7/ /opt/spark)
• Execute with command /opt/spark/bin/spark-shell if you wish to work with Scala or /opt/spark/bin/pyspark if you want to work with Python

Answer 2

For running spark on yarn cluster there is lot of steps to install hadoop and spark and all so i write one blog on it step by step you can install it and run spark shell on yarn see the below link

https://blog.knoldus.com/2016/01/30/spark-shell-on-yarn-resource-manager-basic-steps-to-create-hadoop-cluster-and-run-spark-on-it/

Answer 3

Steps to Install Apache Spark

1) Open Apache Spark Website http://spark.apache.org/

2) Click on Downloads Tab a new Page will get open

3) Choose Pre-built for Hadoop 2.7 and later

4) Choose Direct Download

5) Click on Download Spark: spark-2.0.2-bin-hadoop2.7.tgz and save it on your desired location.

6) Go to the Downloaded Tar file and Extract it.

7) Again Extract the spark-2.0.2-bin-hadoop2.7.tar [File name will differ as version changes] to generate spark-2.0.2-bin-hadoop2.7 folder

8) Now open Shell Prompt and go to the bin directory of spark-2.0.2-bin-hadoop2.7 folder [Folder name will differ as version changes ]

9) Execute command spark-shell.sh

You will be in Spark Shell you can execute the spark commands

https://spark.apache.org/docs/latest/quick-start.html <-- Quick start Guide from spark

Hope this Helps!!!