CODEWITHSUNDEEP
python-3.xaws-lambdaamazon-iam
asur
asur

Reputation: 1879

Python3.7 script to export CloudWatch logs to S3

I am using below code to copy CloudWatch logs to S3:-

import boto3
import collections
from datetime import datetime, date, time, timedelta

    region = 'eu-west-1'

    def lambda_handler(event, context):
        yesterday = datetime.combine(date.today()-timedelta(1),time())
        today = datetime.combine(date.today(),time())
        unix_start = datetime(1970,1,1)
        client = boto3.client('logs')
        response = client.create_export_task(
            taskName='Export_CloudwatchLogs',
            logGroupName='/aws/lambda/stop-instances',
            fromTime=int((yesterday-unix_start).total_seconds() * 1000),
            to=int((today -unix_start).total_seconds() * 1000),
            destination='bucket',
            destinationPrefix='bucket-{}'.format(yesterday.strftime("%Y-%m-%d"))
        )
        return 'Response from export task at {} :\n{}'.format(datetime.now().isoformat(),response)

I gave below policy to role:-

policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams",
        "logs:CreateExportTask",
        "logs:DescribeExportTasks",
        "logs:DescribeLogGroups"
    ],
      "Resource": [
        "arn:aws:logs:*:*:*"
    ]
  }
 ]
}
EOF

2nd policy:-

policy = <<EOF
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Action": [
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:GetBucketAcl"
          ],
          "Effect": "Allow",
          "Resource": ["arn:aws:s3:::${var.source_market}-${var.environment}-${var.bucket}/*"],
          "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } }
        }
      ]
    }
    EOF

I am getting below error if I execute this in AWS console:-

{ "errorMessage": "An error occurred (InvalidParameterException) when calling the CreateExportTask operation: GetBucketAcl call on the given bucket failed. Please check if CloudWatch Logs has been granted permission to perform this operation.", "errorType": "InvalidParameterException"

I have referred many blocks after appending role with appropriate policies.

Upvotes: 2

Views: 4224

Answers (3)

Asim
Asim

Reputation: 543

It seems like an issue with s3 bucket permissions. You need to attach this policy to your s3 bucket. Please amend the policy by changing the bucket name and aws region for cloudwatch.

{
"Version": "2012-10-17",
"Statement": [
  {
      "Action": "s3:GetBucketAcl",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-exported-logs",
      "Principal": { "Service": "logs.us-west-2.amazonaws.com" }
  },
  {
      "Action": "s3:PutObject" ,
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
      "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } },
      "Principal": { "Service": "logs.us-west-2.amazonaws.com" }
  }
]}

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasksConsole.html

Upvotes: 2

Joaquin Mendoza
Joaquin Mendoza

Reputation: 1

I had the same error, the issue was that I put on "destination" parameter something like bucket/something while on the policy I just had bucket, so removing the something prefix on the parameter fixed the problem, so check that the policy and the parameter match.

Upvotes: 0

lonnix
lonnix

Reputation: 11

Check the encryption settings on your bucket. I had the same problem and it was because I had it set to AWS-KMS. I was getting this error with the same permissions you have and then it started working as soon as I switched the encryption to AES-256

Upvotes: 1

Related Questions