CODEWITHSUNDEEP
amazon-web-servicesamazon-kinesis-firehose
Giles Hunt
Giles Hunt

Reputation: 531

Slow PutRecordBatch on Kinesis Firehose

We're getting really slow PutRecordBatch writes to all our Kinesis firehose Streams in EU-West-1. This has been happenign on and off for several days.

It started as a slow running Lambda function so we've looked at Lambda X-ray to see where the slow-down was occurring and its consistently write connections to put Firehose. Sometimes they succeed straightaway other times they take several seconds or even minutes to complete. We write to multiple Firehose streams and the slowdown seems to be random across the streams (see below).

Lambda X-Ray

Any help greatly appreciated. It feels like we're hitting some kind of limit that is rate throttling our requests???

Upvotes: 4

Views: 2942

Answers (2)

meeza
meeza

Reputation: 704

I faced similar issues and fixed it using Async client : AmazonKinesisFirehoseAsync.

The results are incredible, earlier the put was very inconsistent and was taking upto seconds in some request, but after using the AmazonKinesisFirehoseAsync it is lot lot faster. Following is the code snippet on how to use it:

AmazonKinesisFirehoseAsyncClientBuilder builder = AmazonKinesisFirehoseAsyncClientBuilder.standard().withRegion("us-west-2"); // put your own region
AmazonKinesisFirehoseAsync firehoseAsync = builder.build();
// put request
firehoseAsync.putRecord(putRecordRequest);

To add the x-ray tracing, you can build the client as below:

AmazonKinesisFirehoseAsync firehoseAsync = builder.withRequestHandlers(new TracingHandler(AWSXRay.getGlobalRecorder())).build();

Upvotes: 2

daks999
daks999

Reputation: 21

Yes, you are getting this issue because you are hitting some of the limits.

Check here on the documentation.

Upvotes: -1

Related Questions