CODEWITHSUNDEEP
pythonamazon-web-servicespysparkamazon-emr
tim_xyz
tim_xyz

Reputation: 13581

Cannot access pyspark in EMR cluster jupyter notebook

I've created a spark cluster on EMR. But I'm unable to access pyspark when I open it with a notebook.

Configuration:

enter image description here

Example:

from pyspark import SparkContext

enter image description here

I also cannot access sc which I was under the impression would be available.

sc.list_packages()

NameError: name 'sc' is not defined

I feel like I'm missing something very basic here but I'm completely new to EMR and have spent a bunch of time on this already.

Are there any ideas I should try to debug this?

Upvotes: 2

Views: 759

Answers (1)

tim_xyz
tim_xyz

Reputation: 13581

When I opened my notebook with "JupyterLab" instead of "Jupyter" all libraries were available.

enter image description here

Upvotes: 1

Related Questions