CODEWITHSUNDEEP
pysparkibm-cloudwatson-studioanalytics-engine
Ronen491
Ronen491

Reputation: 71

How to integrate Watson Studio python notebook with an IAE Spark?

I have an IBM Analytics Engine (IAE) instance, added it to my Watson Studio project as an associated service and created an Environment based on it. Then, I created a python notebook and set its environment to the abovementioned.

I ran a simple pyspark script in the notebook and noticed that it uses a local instance instead of the IAE.

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
spark

output:

SparkSession - in-memory
SparkContext

Spark UI
Version
v2.4.5
Master
local[*]
AppName
pyspark-shell

What am I doing wrong?

Upvotes: 0

Views: 128

Answers (1)

Roland Weber
Roland Weber

Reputation: 1965

You're doing nothing wrong. Your code gets sent to IAE for execution. So it is running locally in your IAE instance.

Upvotes: 0

Related Questions