CODEWITHSUNDEEP
spark-streamingamazon-kinesis
EugeneMi
EugeneMi

Reputation: 3575

Spark streaming batch interval with Kenisis

What is the effect of setting batch interval when creating the streaming context

new StreamingContext(spark.sparkContext, batchInterval)

According to this Amazon blog the Kinesis batch interval is hard coded to 1s.

Upvotes: 0

Views: 556

Answers (1)

BigDataGuru
BigDataGuru

Reputation: 31

The the Kinesis batch interval mentioned in the Blog is the interval at which the receiver reads data from a stream, which is by default set at 1 second. This interval just decides input rate of the receiver.

The batchInterval provided while creating the StreamingContext divides the input records into batches of given interval, to be processed by spark streaming.

For example if you have single Kinesis receiver and your batchInterval is 10 seconds then receiver would be able to read up to 10000 records in 10 second, that is reading 1000 records per second interval from the Kinesis stream. So your that streaming batch will include 10000 records.

Upvotes: 1

Related Questions