Config settings for S3 sink connector

Question

I am new to S3 sink connector, I am trying to setup s3 connector for my project. I have few doubts:

• What is the use of flush.size in config? what if I give the very large number(2147483647) for it, then how would it work?
• rotate.interval.ms - How does this work if I give its value as 1 hour ?
• rotate.schedule.interval.ms - If I use this setting would I get duplicate records?

Thanks in advance!

Answer 1

flush size determines an upper bound on the number of records you expect in files; if you set it large, then you'll either run out of memory/disk space on the Connect workers, or you'll end up one large file per partition or rotation interval

The two settings operate on different intervals, and shouldn't be combined. If you rotate every hour, you should expect the records (who's buffer size would be less than the flush size at that time) to be flushed that often

For next question, from documentation

setting rotate.schedule.interval.ms is nondeterministic and will invalidate exactly-once guarantees