CODEWITHSUNDEEP
amazon-web-servicesamazon-kinesisamazon-kinesis-firehoseamazon-cloudwatchlogs
Raju
Raju

Reputation: 169

Amazon Kinesis Firehose as a cloudwatch logs consumer

My requirement is to collect the VPC flow logs from AWS. I have created a VPC flow log and associated it to cloud watch log group "Flow log group". From CloudWatch logs subscription I thought of subscribing it through the AWS Kinesis firehose, but in the subscription list I could not find the "Kinesis Firehose", Only "Lambda" and "ElasticSearch Service". Could you please help me on this.

Cannot we subscribe the CloudWatch logs using AWS Kinesis firehose.

Upvotes: 4

Views: 3677

Answers (1)

Sean Summers
Sean Summers

Reputation: 2694

A Firehose arn is a valid subscription destination for CloudWatch Logs, but it is not possible to set one with the console, only with API or CloudFormation.

Most examples I have found use the console's LogGroup option 'Stream to AWS Lambda' to feed a AWS Lambda that forwards to Amazon Kinesis Firehose, such as in How to Visualize and Refine Your Network’s Security by Adding Security Group IDs to Your VPC Flow Logs.

The Using CloudWatch Logs Subscription Filters chapter of the Amazon CloudWatch Logs User Guide has a Firehose Example (Example 3) that might be a better fit for your use case.

Essentially, the path is Amazon CloudWatch LogGroup SubscriptionFilter -> Amazon Kinesis Firehose Delivery Stream -> Amazon S3 Bucket and Prefix

Fun Fact: Amazon CloudWatch Logs data is already GZIPped when delivered to Firehose.

Upvotes: 8

Related Questions