CODEWITHSUNDEEP
pythonpysparkpycharm
KKS
KKS

Reputation: 253

PYCHARM Error-- java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified

I am getting the below error while running a pyspark program on PYCHARM, Error:

java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified ......

The interpreter is recognizing the python.exe file and I have added the Content root in project structure.

I got a similar issue while running the same program before in on windows command prompt and solved it using What is the right way to edit spark-env.sh before running spark-shell?

Upvotes: 24

Views: 27671

Answers (4)

HappilyCoding
HappilyCoding

Reputation: 477

If you are working with a virtual environment, you might need to point the environmental variable to the python of the virtual environment.

For example, in my case the correct path was: os.environ['PYSPARK_PYTHON'] = '...venv\scripts\python.exe'

Upvotes: 0

Nuka Tejeswara Rao
Nuka Tejeswara Rao

Reputation: 271

create an environment variable PYSPARK_PYTHON with value 'python' or the path to your respective python executable.

Upvotes: 26

Cassius Clay
Cassius Clay

Reputation: 332

  1. Go to Environmental variable and within System variable set a new variable as PYSPARK_PYTHON and value as python

PYSPARK_PYTHON=python

  1. Add below codebits to your pyspark code
import os
import sys
from pyspark import SparkContext
os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable

Upvotes: 10

elyptikus
elyptikus

Reputation: 1148

Before creating your spark session, set the following environment variables in your code:

import os
import sys
from pyspark.sql import SparkSession

os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
spark = SparkSession.builder.getOrCreate()

Upvotes: 44

Related Questions