Garion S.
Reputation: 129
CloudFormation add trigger for existing s3 bucket
My goal is to pack my lambda code which is invoked on each image upload to bucket, into CloudFormation template. I achieved so far to create new resources, and trigger from scratch, but I have existing bucket to which I need to add trigger and get errors in 2 cases:
- when I put creation of lambda and trigger configuration in one template and try to create stack as new resources - it says that bucket already exists
- when I move trigger to new file - and first create new resources like in 1., then I import existing resources into the stack - I get:
There was an error creating this change set
You have modified resources [ScaleImages, ScaleImagesRole] in your template that are not being imported. Update, create or delete operations cannot be executed during import operations.
My templates looks like:
- lambda creation - new lambda and role - create stack with new resources
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"ScaleImages": {
"Type": "AWS::Lambda::Function",
"DeletionPolicy": "Retain",
"Properties": {
"FunctionName": "ScaleImages",
"Handler": "index.handler",
"Role": {
"Fn::GetAtt": [
"ScaleImagesRole",
"Arn"
]
},
"Code": {
"S3Bucket": "example-test",
"S3Key": "example-resize.zip"
},
"Runtime": "nodejs12.x",
"MemorySize": 1024,
"Timeout": 300
}
},
"ScaleImagesRole": {
"Type": "AWS::IAM::Role",
"DeletionPolicy": "Retain",
"Properties": {
"RoleName": "ScaleImagesRole",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Path": "/",
"Policies": [
{
"PolicyName": "AWSLambdaBasicExecutionRole",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}
},
{
"PolicyName": "AmazonS3FullAccess",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::example-test",
"arn:aws:s3:::example-test/*",
"arn:aws:s3:::example-test-output",
"arn:aws:s3:::example-test-output/*"
]
}
]
}
}
]
}
}
}
}
- adding trigger - bucket exists - Import resources
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"PutOriginalImage": {
"Type": "AWS::S3::Bucket",
"DeletionPolicy": "Retain",
"Properties": {
"BucketName": "example-test",
"NotificationConfiguration": {
"LambdaConfigurations": [
{
"Event": "s3:ObjectCreated:Put",
"Filter": {
"S3Key": {
"Rules": [
{
"Name": "prefix",
"Value": "original2/"
}
]
}
},
"Function": {
"Fn::GetAtt": [
"ScaleImages",
"Arn"
]
}
}
]
}
}
}
}
}
In last one I tried also "Function": "ScaleImages", but in both cases I had same error about:
modified resources [ScaleImages, ScaleImagesRole] in your template
Can somebody shed some light on what I'm doing wrong?
Upvotes: 6
Views: 4437
Answers (2)
samtoddler
Reputation: 9665
A slightly different approach that gets you going in one shot without following 3 steps. I have been through some tough times while importing existing resources in Cloudformation, I would handle the complexity in the lambda via a custom resource
s3 = boto3.resource('s3')
def lambda_handler(event, context):
print("Received event: " + json.dumps(event, indent=2))
responseData={}
try:
if event['RequestType'] == 'Delete':
print("Request Type:",event['RequestType'])
Bucket=event['ResourceProperties']['Bucket']
delete_notification(Bucket)
print("Sending response to custom resource after Delete")
elif event['RequestType'] == 'Create' or event['RequestType'] == 'Update':
print("Request Type:",event['RequestType'])
LambdaArn=event['ResourceProperties']['LambdaArn']
Bucket=event['ResourceProperties']['Bucket']
add_notification(LambdaArn, Bucket)
responseData={'Bucket':Bucket}
print("Sending response to custom resource")
responseStatus = 'SUCCESS'
except Exception as e:
print('Failed to process:', e)
responseStatus = 'FAILURE'
responseData = {'Failure': 'Something bad happened.'}
cfnresponse.send(event, context, responseStatus, responseData)
def add_notification(LambdaArn, Bucket):
bucket_notification = s3.BucketNotification(Bucket)
response = bucket_notification.put(
NotificationConfiguration={
'LambdaFunctionConfigurations': [
{
'LambdaFunctionArn': LambdaArn,
'Events': [
's3:ObjectCreated:*'
]
}
]
}
)
print("Put request completed....")
def delete_notification(Bucket):
bucket_notification = s3.BucketNotification(Bucket)
response = bucket_notification.put(
NotificationConfiguration={}
)
print("Delete request completed....")
Full template and solution can be found here
Note : There is alaready an open issue on AWS CloudFormation Repo on github . Orignally came from the serverless folks
Upvotes: 1
Marcin
Reputation: 238937
You have to do it in stages:
1. Create new stack
No bucket yet, just stack with your function and lambda permissions which you are missing.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"ScaleImages": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "ScaleImages",
"Handler": "index.handler",
"Role": {
"Fn::GetAtt": [
"ScaleImagesRole",
"Arn"
]
},
"Code": {
"S3Bucket": "example-test",
"S3Key": "example-resize.zip"
},
"Runtime": "nodejs12.x",
"MemorySize": 1024,
"Timeout": 300
}
},
"ScaleImagesRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": "ScaleImagesRole",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Path": "/",
"Policies": [
{
"PolicyName": "AWSLambdaBasicExecutionRole",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:s3:::example-test",
"arn:aws:s3:::example-test/*",
"arn:aws:s3:::example-test-output",
"arn:aws:s3:::example-test-output/*"
]
}
]
}
},
{
"PolicyName": "AmazonS3FullAccess",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
}
]
}
},
"s3Permission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Fn::GetAtt": [
"ScaleImages",
"Arn"
]
},
"Action": "lambda:InvokeFunction",
"Principal": "s3.amazonaws.com",
"SourceAccount": {
"Ref": "AWS::AccountId"
}
}
}
}
}
2. Import bucket to existing stack
Use Import resources into stack option and upload stack using this template. It adds bucket, but no notifications yet
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"ScaleImages": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "ScaleImages",
"Handler": "index.handler",
"Role": {
"Fn::GetAtt": [
"ScaleImagesRole",
"Arn"
]
},
"Code": {
"S3Bucket": "example-test",
"S3Key": "example-resize.zip"
},
"Runtime": "nodejs12.x",
"MemorySize": 1024,
"Timeout": 300
}
},
"ScaleImagesRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": "ScaleImagesRole",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Path": "/",
"Policies": [
{
"PolicyName": "AWSLambdaBasicExecutionRole",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:s3:::example-test",
"arn:aws:s3:::example-test/*",
"arn:aws:s3:::example-test-output",
"arn:aws:s3:::example-test-output/*"
]
}
]
}
},
{
"PolicyName": "AmazonS3FullAccess",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
}
]
}
},
"s3Permission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Fn::GetAtt": [
"ScaleImages",
"Arn"
]
},
"Action": "lambda:InvokeFunction",
"Principal": "s3.amazonaws.com",
"SourceAccount": {
"Ref": "AWS::AccountId"
}
}
}
,
"PutOriginalImage": {
"Type": "AWS::S3::Bucket",
"DeletionPolicy": "Retain",
"Properties": {
"BucketName": "example-test"
}
}
}
}
3. Update stack
Update stack by adding notifications to the bucket. Use the following template:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"ScaleImages": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "ScaleImages",
"Handler": "index.handler",
"Role": {
"Fn::GetAtt": [
"ScaleImagesRole",
"Arn"
]
},
"Code": {
"S3Bucket": "example-test",
"S3Key": "example-resize.zip"
},
"Runtime": "nodejs12.x",
"MemorySize": 1024,
"Timeout": 300
}
},
"ScaleImagesRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": "ScaleImagesRole",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Path": "/",
"Policies": [
{
"PolicyName": "AWSLambdaBasicExecutionRole",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:s3:::example-test",
"arn:aws:s3:::example-test/*",
"arn:aws:s3:::example-test-output",
"arn:aws:s3:::example-test-output/*"
]
}
]
}
},
{
"PolicyName": "AmazonS3FullAccess",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
}
]
}
}
,
"s3Permission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Fn::GetAtt": [
"ScaleImages",
"Arn"
]
},
"Action": "lambda:InvokeFunction",
"Principal": "s3.amazonaws.com",
"SourceAccount": {
"Ref": "AWS::AccountId"
}
}
},
"PutOriginalImage": {
"Type": "AWS::S3::Bucket",
"DeletionPolicy": "Retain",
"Properties": {
"BucketName": "example-test",
"NotificationConfiguration": {
"LambdaConfigurations": [
{
"Event": "s3:ObjectCreated:Put",
"Filter": {
"S3Key": {
"Rules": [
{
"Name": "prefix",
"Value": "original2/"
}
]
}
},
"Function": {
"Fn::GetAtt": [
"ScaleImages",
"Arn"
]
}
}
]
}
}
}
}
}
Upvotes: 2
Related Questions
- How to add triggers for a AWS Lambda function created using a CloudFormation template?
- Cloudformation template to trigger Lambda on S3 event
- Adding S3 trigger to Lambda function using CloudFormation
- aws lambda - How to add s3 trigger using cloudformation if the s3 bucket is created manually
- Enable Lambda function to an S3 bucket using cloudformation
- add S3 trigger on a Lambda function with cloudformation yaml
- Cloudformation - Lambda Trigger on S3 Event? (Bucket Already Exists)
- Cloudformation Trigger Event for Existing bucket
- executing lambda on s3 bucket ObjectCreated event in cloudformation
- Updating an AWS CloudFormation with a custom trigger for Lambda