CODEWITHSUNDEEP
amazon-kinesisamazon-kcl
user1846749
user1846749

Reputation: 2535

kinesis client worker logic

I want to understand when does IRecordProcessor's processRecords method is invoked from worker. If my earlier call to processRecords is not yet completed will worker call next processRecords ? Will worker start fetching new records from kinesis or will it wait till current records finish execution.

Basically I want to wait for a long time if processRecords gets some exception while saving records in external db since db was down or some other error . So want to confirm there won't be any issue in that if worker does not start fetching new records until earlier are finished processing ?

Upvotes: 2

Views: 1892

Answers (1)

az3
az3

Reputation: 3649

Excerpt from other questions:

The application (with the help of KCL) will continue to poll "Shard Iterator" in the background, thus you will be notified about the new data when it comes.

Source: https://stackoverflow.com/a/35582161/1622134

And also, by "worker" you mean a "Worker" thread in the application; which is a runnable.

Each shard is processed by exactly one KCL worker and has exactly one corresponding record processor, so you never need multiple instances to process one shard. See the Worker.java class in KCL source.

Source: https://stackoverflow.com/a/34509567/1622134

To answer you question, you can that it in your processRecords implementation. While processing records, use a try-catch block and write checkpoint to DynamoDB if and only if the try part succeeds. That way; if there is an error while writing to external db, you will not lose records and upon restart. You should also save those record data (which cannot be inserted in the db) to another place to process later.

Also see this answer: https://stackoverflow.com/a/32517002/1622134

Upvotes: 1

Related Questions