CODEWITHSUNDEEP
scalaapache-sparkapache-kafkaspark-streamingspark-structured-streaming
samthebest
samthebest

Reputation: 31515

Limit kafka batch size when using Spark Structured Streaming

We have some historical data queued up on our topics, we don't want to process all this data in a single batch as that is harder to do (and if it fails it has to start again!).

Also, knowing how to control the batch size would be quite helpful in tuning jobs.

When using DStreams the way to control the size of the batch as exactly as possible is Limit Kafka batches size when using Spark Streaming

The same approach i.e. setting maxRatePerPartition and then tuning batchDuration is extremely cumbersome but works with DStream it doesn't work at all with Structured Streaming.

Ideally I'd like to know of a config like maxBatchSize and minBatchSize, where I can simply set the number of records I'd like.

Upvotes: 10

Views: 8625

Answers (2)

Raptor0009
Raptor0009

Reputation: 268

If the topic is partitioned and all the partitions has messages, the minimum messages you can take is equal to the number of partitions in the topic. (ie) it takes 1 record per partition if it has data, if only one partition has data then the minimum record you can take is 1. If the topic is not partitioned you can take 1 record minimum and anything as maximum.

Upvotes: 0

10465355
10465355

Reputation: 4621

This config optionmaxOffsetsPerTrigger:

Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume.

Note that if you have a checkpoint directory with start and end offsets, then the application will process the offsets in the directory for the first batch, thus ignoring this config. (The next batch will respect it).

Upvotes: 12

Related Questions