CODEWITHSUNDEEP
unit-testingapache-sparkpysparkcontinuous-integrationtox
whatsinthename
whatsinthename

Reputation: 2157

Tox with pyspark

We implemented unit tests for the Pyspark scenarios. But as tox creates the runtime environment and installs the required packages, every time I run tox, it has to install pyspark which takes around 3 minutes everytime. So how can I avoid this long package installation runtime so that my test stage in CI/CD pipeline will be executed faster?

Is there any other alternative for this? I did my due diligence but couldn't find anything relevant to my problem.

Upvotes: 1

Views: 129

Answers (1)

404
404

Reputation: 382

You can install Spark (bundled with PySpark) and add a .pth file to each of the Python installs' site-packages directories that references the python directory of the Spark install. I've done that with some success, but different Python versions are compatible with different Spark versions so you'll need multiple depending on your tox environments.

Upvotes: 0

Related Questions