CODEWITHSUNDEEP
apache-sparkpysparkpycharm
Karthik Thirumuruganathan
Karthik Thirumuruganathan

Reputation: 1

Unable to set Python 3 as default version for Apache spark in MacOS

I have been trying to keep Python 3 as default version for Spark in my Macbook, But its not setting up. I am unable to run my programs also due to that .

This is the error I am getting in Pycharm

"Exception: Python in worker has different version 2.7 than that in driver 3.7, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

Can you please help me on this?

Spark version in Terminal

Error as shown in Pycharm

Upvotes: 0

Views: 3119

Answers (1)

Kevin Smith
Kevin Smith

Reputation: 43

Just set the environment variable:

export PYSPARK_PYTHON=python3

in case you want this to be a permanent change add this line to pyspark script. If python3 is not accessible, you need to pass path to it instead.

Upvotes: 4

Related Questions