CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
user1888243
user1888243

Reputation: 2691

How is SparkSession Is Shared in Multiuser Projects?

I'm new to Spark SQL, but understand how to create a SparkSession, load data files, create tables out of the loaded data files, and query the loaded tables for a one user application.

Now, assume, I want to enable a multi-user environment where users can go to a website and run queries against tables I have loaded in Spark SQL.

How do I accomplish this? I obviously don't want to load tables for each user. How is it done in real world applications? Do I simply make SparkSession singleton, and then share this SparkSession between users, and let them run queries on this SparkSession?

Upvotes: 0

Views: 608

Answers (1)

Assaf Mendelson
Assaf Mendelson

Reputation: 13001

There are a couple of already existing tools which can help you.

Take a look at https://github.com/spark-jobserver/spark-jobserver or https://github.com/cloudera/livy

Upvotes: 2

Related Questions