CODEWITHSUNDEEP
pythonapache-sparkpysparkpip
clstaudt
clstaudt

Reputation: 22438

pyspark: Could not find valid SPARK_HOME

After a fresh installation of pyspark via pip install pyspark I get the following errors:

> pyspark
Could not find valid SPARK_HOME while searching ['/Users', '/usr/local/bin']
/usr/local/bin/pyspark: line 24: /bin/load-spark-env.sh: No such file or directory
/usr/local/bin/pyspark: line 77: /bin/spark-submit: No such file or directory
/usr/local/bin/pyspark: line 77: exec: /bin/spark-submit: cannot execute: No such file or directory
> spark-shell
Could not find valid SPARK_HOME while searching ['/Users', '/usr/local/bin']
/usr/local/bin/spark-shell: line 57: /bin/spark-submit: No such file or directory

What is a valid SPARK_HOME, how do I set it, and why is there no default that works?

I've seen instructions on how to manually set the environment variable after a manual installation of spark, but I am wondering how to set it in this case after using pip to install pyspark.

I've installed only spark via brew install apache-spark, and the spark-shell that comes out of that installation works just fine out of the box. After installing pyspark after that, I get the messages above. Confusing.

Upvotes: 6

Views: 7841

Answers (2)

kepy97
kepy97

Reputation: 1074

You can run below command if you already installed apache-pyspark through brew.

brew link --overwrite apache-spark

Upvotes: 0

clstaudt
clstaudt

Reputation: 22438

> brew install apache-spark

actually already provides a working pyspark shell. It is not necessary to additionally 

> pip install pyspark

and in fact this breaks the installation for me.

Upvotes: 6

Related Questions