Cannot get spark/pyspark working on macOS Mojave

Question

Have a new MacBook w/ macOS Mojave but cannot get spark/pyspark working. I have done the following:

• Downloaded the latest Apache Spark from https://spark.apache.org/downloads.html
• Unzipped it in my home directory (/Users/myname/)
• Renamed the folder spark
• Checked in the Finder window that spark was there, and the folder contained python, bin, and bin/spark-shell subfolders
• Got the ability to pip install packages with: curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py >> python get-pip.py in the terminal >> Checked that it was installed by doing import pip in the python shell >> sudo easy_install pip in the regular terminal
• pip install pyspark, which worked
• Created both a .bashrc and .bash_profile files in the Users/myname/ path
• I tried the following versions of the commands in these bash files:

export SPARK_HOME="/Users/myname/spark/python"
export PATH="$SPARK_HOME/bin:$PATH"

export SPARK_HOME=/Users/myname/spark/python
export PATH=$SPARK_HOME/bin:$PATH

export SPARK_HOME=/Users/myname/spark
export PATH=$SPARK_HOME/bin:$PATH

export SPARK_HOME=~/spark
export PATH=$SPARK_HOME/bin:$PATH

Different installation articles say I should just be able to do pyspark in the terminal, or spark/bin/spark-shell or just 'bin/spark-shell`, I just keep getting versions of the message

-bash: pyspark: command not found

Any advice?

Answer 1

If you've installed pyspark from pip, you don't need the standalone installation. You just need to fix you env vars. First, find the location of your site packages:

python -c "import site; print(site.getsitepackages())"

This is probably something like this /usr/lib/pythonX.Y/site-packages

Then, add the exports to your bash profile and source it.

echo "export SPARK_HOME=/usr/lib/pythonX.Y/site-packages/pyspark" >> ~/.bash_profile
echo "export PATH=$SPARK_HOME/bin:$PATH" >> ~/.bash_profile
source ~/.bash_profile