CODEWITHSUNDEEP
pythonscalaapache-spark
London guy
London guy

Reputation: 28032

How do you integrate this new Spark libary for python?

I am using spark with python using the pyspark command.

I came across this new library

https://github.com/databricks/spark-csv

which I want to integrate into my spark program written in python.

I have downloaded the complete codebase shown in the above link but not sure how to proceed from there. Should I just copy its contents somewhere?

Any guidance would help as I am still learning scala and spark and want to use the above library code.

Upvotes: 0

Views: 40

Answers (1)

zero323
zero323

Reputation: 330393

Just to repeat information from the README file. To use spark-csv in your script you should use --packages option for each execution of spark-submit or pyspark shell.

Upvotes: 2

Related Questions