CODEWITHSUNDEEP
macosapache-sparkpyspark
Rich Gray
Rich Gray

Reputation: 201

Can not run pyspark command from any directory on my Mac after installation of Apache Spark

I have installed spark on my Mac, following the instructions in the book: "Apache Spark in 24 Hours". When I am in the spark directory, I am able to run pyspark by using the command:

./bin/pyspark

To install spark I created the env variable:

export SPARK_HOME=/opt/spark

Added it to the PATH:

export PATH=$SPARK_HOME/bin:$PATH

The book says that I should be able to run the "pyspark" or the "spark-shell" command from any directory, but it doesn't work:

pyspark: command not found

I followed instructions on similar questions asked by others on here:

I set my JAVA_HOME env variable:

export JAVA_HOME=$(/usr/libexec/java_home)

I also ran the following commands:

export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.9-src.zip:$PYTHONPATH

When I run the env command this is the output:

SPARK_HOME=/opt/spark
TERM_PROGRAM=Apple_Terminal
SHELL=/bin/bash
TERM=xterm-256color
TMPDIR=/var/folders/hq/z0wh5c357cbgp1dh33lfhjj40000gn/T/
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.fJdtLqZ7dN/Render
TERM_PROGRAM_VERSION=361.1
TERM_SESSION_ID=A8BD2144-72AD-402C-A591-5C8A43DD398B
USER=richardgray
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.cQeqaF2v1z/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PATH=/opt/spark/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:  /Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/heroku/bin:/Users/richardgray/.rbenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
PWD=/Users/richardgray
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home
LANG=en_GB.UTF-8
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
SHLVL=1
HOME=/Users/richardgray
PYTHONPATH=/opt/spark/python/lib/py4j-0.9-src.zip:/opt/spark/python/:
LOGNAME=richardgray
_=/usr/bin/env

Is there something I am missing? Thanks in advance.

Upvotes: 2

Views: 7036

Answers (1)

Yaron
Yaron

Reputation: 10450

You wrote that

When I am in the spark directory, I am able to run pyspark by using the command: ./bin/pyspark

You created export SPARK_HOME=/opt/spark

Can you please confirm that spark directory is indeed /opt/spark ?

If you execute spark from /Users/richardgray/opt/spark/bin please set:

export SPARK_HOME=/Users/richardgray/opt/spark

followed by:

export PATH=$SPARK_HOME/bin:$PATH

Note: If it solve your problem, you'll need to add those two exports to your login scripts (e.g. .profile) so the path will be set automatically

Upvotes: 3

Related Questions