CODEWITHSUNDEEP
amazon-web-serviceselasticsearchloggingaws-lambdaarchitecture
MyCoolAwsomeName
MyCoolAwsomeName

Reputation: 71

AWS buffer logs in CloudWatch before streaming to lambda

Is there a way to buffer X log messages from a CloudWatch log group and only then stream it to a lambda function? I'll elaborate:

I have an app that I registered it's CloudWatch logs to stream to a lambda function which formats the logs and pushes them to Elastic Search.

So the flow is the following: (app logs) -> (CloudWatch) -->(Lambda)-->(Elastic Search)

My problem is that my lambda function is invoked very often (most of the time single log message) and bombards ES with write requests, I would like to write the logs in bulks, i.e wait until 30 new logs and then invoke the lambda for the 30 logs bulk.

The only way I found to achieve this is to use Kinesis and Firehose but those services cost extra and I want to avoid this.

Are there any other alternatives to achieve this without using something like LogStash? I am assuming this is a very common usage so there must be some easy way to solve this.

Thanks,

Upvotes: 0

Views: 548

Answers (2)

bob Ditusa
bob Ditusa

Reputation: 21

I was wondering what you ended up doing in the situation. I believe if you use functionbeats you can not use aws ES you have to create it manually.

Upvotes: 0

Val
Val

Reputation: 217424

I would investigate Functionbeat whose main goal is to stream Cloudwatch logs (among others) to ES. Extremely easy to deploy and operate, no fiddling with Lambda code, etc. A MUST if you're evolving in the AWS environment yet still want to leverage ES as a log engine

Upvotes: 1

Related Questions