CODEWITHSUNDEEP
pysparkapache-spark-sqlspark-streamingspark-structured-streaming
Naresh Tambekar
Naresh Tambekar

Reputation: 23

Temporary View in Spark Structure Streaming

In ForeachBatch Function Structured Straming I want to create Temporary View of the Dataframe Received in the Micro Batch

func(tabdf, epoch_id):
    tabaDf.createOrReplaceView("taba")

But I am getting below error:

org.apache.spark.sql.streaming.StreamingQueryException: Table or view not found: taba
Caused by: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'taba' not found

Please anyone help me to resolve this issue.

Upvotes: 0

Views: 4221

Answers (1)

zsxwing
zsxwing

Reputation: 20816

A streaming query uses its own SparkSession which is cloned from the SparkSession that starts the query. And the DataFrame provided by foreachBatch is created from the streaming query's SparkSession. Hence you cannot access temp views using the original SparkSession.

One workaround is using createGlobalTempView/createOrReplaceGlobalTempView to create a global temp view. Please note that global temp views are tied to a system preserved database global_temp, and you need to use the qualified name to refer a global temp, such as SELECT * FROM global_temp.view1.

Upvotes: 10

Related Questions