CODEWITHSUNDEEP
pysparkjupyter-notebookspark-submit
Q Bit
Q Bit

Reputation: 91

Pyspark: Error executing Jupyter command while running a file using spark-submit

I am able to run pyspark and run a script on Jupyter notebook. But when I try to run the file from terminal using spark-submit, getting this error:

Error executing Jupyter command file path [Errno 2] No such file or directory

Can anyone help me what's wrong with my config?

I'm using Python 2.7 and Spark 1.6

Upvotes: 9

Views: 5391

Answers (2)

Rich
Rich

Reputation: 1

@Shubham's solution resolved my problem: "Jupyter command jupyter-spark_simple.py not found."

unset PYSPARK_DRIVER_PYTHON

Upvotes: 0

Shubham Sharma
Shubham Sharma

Reputation: 2793

These problem occurs because you have set jupyter to run pyspark script . Now you should unset PYSPARK_DRIVER_PYTHON variable command for the same is $ unset PYSPARK_DRIVER_PYTHON then try spark-submit again ./bin/spark-submit examples/src/main/python/streaming/network_wordcount.py localhost 9999.

Upvotes: 29

Related Questions