CODEWITHSUNDEEP
pythonideapache-spark
user2585578
user2585578

Reputation: 415

How to use the pyspark module in python script?

I downloaded the pyspark module, and I would like to use it in my IDE (Spyder). But it is not working at all. I typed export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH to include it in python path, but still "no module named pyspark" is the message I get. Is it always that I have to use pyspark console in order to use the spark module? Is there no way to use the module directly in other python IDE?

Upvotes: 0

Views: 681

Answers (1)

Prince Bhatti
Prince Bhatti

Reputation: 5031

First download sources from spark website. Then download and install sbt from this website.Now go to sbt folder in spark source downloaded. Run sbt file with sbt package installed earlier. It will start building and will take around 30 minutes to complete.For more info,goto this link. Remember, you should have latest version of JDK installed in your PC/Laptop. Get JDK from here.After building , follow the commands given in spark website to create pyspark PYTHONPATH and environment. You can do this on spyder also.

Upvotes: 0

Related Questions