CODEWITHSUNDEEP
amazon-web-servicesamazon-dynamodb-streamsamazon-kcl
Mason Chester
Mason Chester

Reputation: 43

AWS KCL worker on shutdown populates the lease table with more leases then shards exist

Currently I am in the middle of building a dynamodb streaming service using AWS KCL library specifically the dynamodb streams adapter (version 1.6.0), in my initial testing implementation I am seeing some weird interactions between the lease table and my worker.

When running the aws cli command aws dynamodbstreams --describestreams the stream currently has only 15 shards.

The lease table will start out with more leases than shards around 105, it will work its way all the way down to holding on to 15 leases. If the application is then shutdown at this stage my logging will output shutdown reason and for those 15 leases the workers shutdown reason is ZOMBIE.

After shutdown is complete the table is populated with about 97 records and then the proceeding runs break. The only way that I have been able to solve this was deleting the table and letting it re run but it will repeat.

This is an example of my shutdown method.

    public void shutdown(ShutdownInput shutdownInput) {
        log.info("Shutting down, reason: " + shutdownInput.getShutdownReason());
        try {
            if (shutdownInput.getShutdownReason() == ShutdownReason.TERMINATE && prevRecord != null) {
                shutdownInput.getCheckpointer().checkpoint(prevRecord);
            }else if (shutdownInput.getShutdownReason() != ShutdownReason.ZOMBIE){
                shutdownInput.getCheckpointer().checkpoint();
            }
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        }
    }

I am expecting that when a worker is shutdown it picks the lease backup at the spot previously checkpoint.

UPDATE:

I have sense figured out the issue, it turns out that running your Worker in a normal Thread; calling .shutdown() on the worker causes the worker to to think that all of its RecordProcessor instances are acting like a ZOMBIEs (not refreshing its lease). The solution was to place the worker in a single thread ExecutorService.

Upvotes: 1

Views: 95

Answers (0)

Related Questions