darkknight444
Reputation: 546
How does back pressure property work in Spark Streaming?
I have a CustomReceiver which receives a single event(String).The received single event is used during spark application's run time to read data from nosql and to apply transformations.When the processing time for each batch was observed to be greater than batch interval I set this property.
spark.streaming.backpressure.enabled=true
After which I expected the CustomReceiver to not trigger and receive the event when a batch is processing longer than batch window, which didn't happen and still a backlog of batches were being added. Am I missing something here?
Upvotes: 7
Views: 10765
Answers (1)
Related Questions
- How is spark.streaming.kafka.maxRatePerPartition related to spark.streaming.backpressure.enabled incase of spark streaming with Kafka?
- Backpressure in storm
- How Spark Structured Streaming handles backpressure?
- Spark Streaming Kafka backpressure
- How to handle backpressure on databases when using Apache Spark?
- How to enable backpressure in Spark Streaming (using pyspark)
- Avoiding data loss when slow consumers force backpressure in stream processing (spark, aws)
- Understanding backpressure in akka streams Source.queue
- Trying to understand spark streaming flow
- How does Spark streaming and Dstream work?