CODEWITHSUNDEEP
apache-sparkpyspark
James Gan
James Gan

Reputation: 7116

Is it possible to create persistent view in Spark?

I'm learning Spark and found that I can create temp view in Spark by calling one of following pySpark API:

df.createGlobalTempView("people")
df.createTempView("people")
df.createOrReplaceTempView'("people")

Can I create a permanent view to that it became available for every user of my spark cluster? Think this will save people's time if views are already defined for them.

Upvotes: 6

Views: 8567

Answers (2)

Andrei Efimov
Andrei Efimov

Reputation: 78

By paradigm, Spark doesn't have any persistence capabilities since it's a data processing engine but not data warehousing.

If you want to provide some session independent views you need to work with existing Hive deployment or use an approach with Spark owned metastore. For more details please refer Spark doc about Hive interaction.

Upvotes: -4

Alper t. Turker
Alper t. Turker

Reputation: 35219

Yes, but you'll have to use SQL:

spark.sql("CREATE VIEW persistent_people AS SELECT * FROM people")

Upvotes: 6

Related Questions