CODEWITHSUNDEEP
pythonapache-sparkpyspark
Parker
Parker

Reputation: 303

Apache-spark - Error launching pyspark on windows

I am trying to run pyspark on windows. However, I received following error:

c:\spark>pyspark
Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC 
v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
File "C:\spark\bin\..\python\pyspark\shell.py", line 38, in <module>
   SparkContext._ensure_initialized()
File "C:\spark\python\pyspark\context.py", line 259, in _ensure_initialized
   SparkContext._gateway = gateway or launch_gateway(conf)
File "C:\spark\python\pyspark\java_gateway.py", line 80, in launch_gateway
   proc = Popen(command, stdin=PIPE, env=env)
File "C:\Users\shuzhe\Anaconda3\lib\subprocess.py", line 707, in __init__
   restore_signals, start_new_session)
File "C:\Users\shuzhe\Anaconda3\lib\subprocess.py", line 990, in _execute_child
startupinfo)
PermissionError: [WinError 5] Access is denied

I have searched for hours, but I could not find a solution to this problem. Any help is greatly appreciated!

Thanks!

Upvotes: 2

Views: 758

Answers (2)

leeyuiwah
leeyuiwah

Reputation: 7152

Hi I ran into the same problem. I ditched the SPARK_HOME folder (which in my case was C:\spark-2.4.3-bin-hadoop2.7) that I created previously using cygwin tar and tried unpacking the folder again (from spark-2.4.3-bin-hadoop2.7.tgz), but this time using 7zip instead of cygwin tar and then problem went away. Note that with 7zip the unpacking is done in two step, first extract the ...tar from ...tgz, and then extract the whole directory tree from ...tar

Some other discussion on the Internet (e.g. here) suggested that the problem was due to cygwin tar got some permission wrong. This explanation is consistent with what I tested and experienced above.

Upvotes: 1

zlidime
zlidime

Reputation: 1224

If you are using spark 2.1 you will have to switch to spark 2.1.1.There is a bug with python 3.6 and now is resolved in 2.1.1 .

https://issues.apache.org/jira/browse/SPARK-19019\

I had same problem with python 3.6 and 3.5 and spark 2.1 on windows , and I switched to anaconda python 2.7 and now it works fine.

Upvotes: 0

Related Questions