CODEWITHSUNDEEP
apache-sparkspark-structured-streaming
nrvaller
nrvaller

Reputation: 373

In Spark's Structured Streaming is there a Streaming Context equivalent like there is in Spark Streaming?

I am trying to grab batch information of a structured streaming query to eventually log. I am new to Spark in general and I'm confused about accessing the equivalent of a streamingContext from Spark Streaming. Is streamingContext something that is solely to Spark Streaming?

I was trying to do something similar to the following except I haven't been able to register the listener(therefore I'm trying to understand) because I don't have a streamingContext. Am I on a completely off track by trying to have a StreamingContext with Structured Streaming?

In Spark Streaming, is there a way to detect when a batch has finished?

Upvotes: 2

Views: 440

Answers (1)

Molotch
Molotch

Reputation: 475

The StreamingQueryListener has lifetime events of the query, onQueryStarted, onQueryUpdated, onQueryTerminated. OnQueryUpdated is triggered in between batches and report on the last processed batch as far as I understand.

Create a custom listener and attach to the eventbus and you should be good to go.

spark.streams.addListener(customListener)

Upvotes: 2

Related Questions