CODEWITHSUNDEEP
apache-sparkpysparkjupyter-notebookamazon-sagemaker
Ksheth
Ksheth

Reputation: 33

Install spark 3.2 in SageMaker Notebook

SageMaker supports pyspark 2.4 and I want to install latest version of pyspark in Sagemaker. Can you please let me know how can I install latest version of pyspark in SageMaker notebook.

Upvotes: 1

Views: 2041

Answers (1)

durga_sury
durga_sury

Reputation: 1152

I'm assuming you're referring to SageMaker notebook instances.

You can use the magic command %%local to run commands locally on your instance.

For example,

%%local
!pip install --upgrade pyspark

You may need to restart the kernel for the upgraded package, and you can test it like below -

%%local
import pyspark
print("PySpark version: ", pyspark.__version__)

Upvotes: 2

Related Questions