CODEWITHSUNDEEP
pythonapache-sparkpyspark
sam
sam

Reputation: 111

Spark Python error "FileNotFoundError: [WinError 2] The system cannot find the file specified"

I am new to Spark and Python. I have installed python 3.5.1 and Spark-1.6.0-bin-hadoop2.4 on windows.

I am getting the below error when I execute sc = SparkContext("local", "Simple App") from the Python shell:

>>> from pyspark import SparkConf, SparkContext

>>> sc = SparkContext("local", "Simple App")

Traceback (most recent call last):

  File "<pyshell#11>", line 1, in <module>

    sc = SparkContext("local", "Simple App")

  File "C:\spark-1.6.0-bin-hadoop2.4\python\pyspark\context.py", line 112, in __init__

    SparkContext._ensure_initialized(self, gateway=gateway)

  File "C:\spark-1.6.0-bin-hadoop2.4\python\pyspark\context.py", line 245, in _ensure_initialized

    SparkContext._gateway = gateway or launch_gateway()

  File "C:\spark-1.6.0-bin-hadoop2.4\python\pyspark\java_gateway.py", line 79, in launch_gateway

    proc = Popen(command, stdin=PIPE, env=env)

  File "C:\Python35-32\lib\subprocess.py", line 950, in __init__
    restore_signals, start_new_session)

  File "C:\Python35-32\lib\subprocess.py", line 1220, in _execute_child
    startupinfo)

FileNotFoundError: [WinError 2] The system cannot find the file specified

>>>

Upvotes: 9

Views: 20201

Answers (6)

Orlando G
Orlando G

Reputation: 327

I had the same error, in my case was an error in the path of system variables. If you are running it in windows you need to edit the environment variables in Windows.

Here is a video how you can do it.

https://www.youtube.com/watch?v=wt2wM8C2SXA

Upvotes: 0

deerishi
deerishi

Reputation: 647

I had the same error. Make sure SPARK_HOME, JAVA_HOME and the PATH environment variables set correctly

For my machine,

SPARK_HOME: C:\Repo\spark\spark-3.3.1-bin-hadoop3
JAVA_HOME: C:\Program Files\Java\jdk1.8.0_361

In your PATH variable: Append the following

%SPARK_HOME%\bin
%JAVA_HOME%\bin

This is because the SPARK_HOME\bin contains the executables for spark-shell/sql etc.

Upvotes: 1

paul
paul

Reputation: 479

Restart and Run pySpark as an administrator

Upvotes: 2

SureshCS
SureshCS

Reputation: 1045

If the verified the system environment variables and it is still not working, check if your jvm (64 or 32)bit version is compatible with your machine.

Upvotes: 0

Iman Nekooeimehr
Iman Nekooeimehr

Reputation: 111

Check your address to make sure it is written correctly. In my case, I had the address as:

"C:/Users/nekooeimehr/AppData/Local/Programs/Python/Python35-32/spark-1.6.2-bin-hadoop2.4"

while the correct address is:

"C:/Users/nekooeimehr/AppData/Local/Programs/Python/Python35-32/spark-1.6.2-bin-hadoop2.4/spark-1.6.2-bin-hadoop2.4"

Upvotes: 5

bharathi
bharathi

Reputation: 41

You have to set the SPARK_HOME correctly. I have debugged the python scripts and verified. This would work.

Upvotes: 2

Related Questions