CODEWITHSUNDEEP
apache-sparkpysparkjupyter-notebook
Greg Rosen
Greg Rosen

Reputation: 197

How to access Apache PySpark from command line?

I'm taking an online course on Apache PySpark using Jupyter notebooks. In order to easily open the Jupyter notebooks they had me enter these lines of code into my bash profile (I'm using MAC OS):

export SPARK_HOME="(INSERTED MY SPARK DIRECTORY)"
export PATH=$SPARK_HOME/bin:$PATH
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'

I'm not too familiar with Linux and the course didn't explain what these lines of code do. Before I did this, I could access PySpark via the command line by typing "pyspark". But now when I type "pyspark" it opens a jupyter notebook. Now I can't figure out how to access it from the command line. What does this code do and how do I access command line pyspark?

Upvotes: 0

Views: 457

Answers (1)

Piyush Patel
Piyush Patel

Reputation: 1751

Are you using local installation of Pyspark?

You can use https://github.com/minrk/findspark

Install findspark using Anaconda.

First, you add these two lines and it will be able to find pyspark.

import findspark
findspark.init()

import pyspark
sc = pyspark.SparkContext(appName="myAppName")

Upvotes: 1

Related Questions