CODEWITHSUNDEEP
amazon-s3aws-lambdaaws-cloudformationamazon-sqsserverless-framework
Driss Amri
Driss Amri

Reputation: 1805

CloudFormation S3 NotificationConfiguration to SQS

I'm new to CloudFormation and currently trying to send a S3:ObjectCreated to a specific SQS Queue.

The setup is in the Serverless Framework with Resources defined in CloudFormation. The problem is with the NotificationConfiguration with a QueueConfigurations that keeps giving error after error.

The syntax below seems to be correct when looking at the CloudFormation Designer online:

  iamRoleStatements:
    - Effect: Allow
      Action:
        - s3:ListBucket
      Resource:
        Fn::Join:
          - ""
          - - "arn:aws:s3:::"
            - Ref: LabelBucket

    - Effect: Allow
      Action:
        - s3:GetObject
        - s3:PutObject
      Resource:
        Fn::Join:
          - ""
          - - "arn:aws:s3:::"
            - Ref: LabelBucket
            - "/*"

    - Effect: Allow
      Action:
        - SQS:SendMessage
      Resource:
        Fn::Join:
          - ""
          - - "arn:aws:s3:::"
            - Ref: LabelBucket

resources:
  Resources:
    LabelRequestQueue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: label-generate-request

    LabelResponseQueue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: label-generate-response

    LabelBucket:
      Type: AWS::S3::Bucket
      DependsOn:
          - LabelResponseQueue
      Properties:
        BucketName: generation-bucket
        NotificationConfiguration:
          QueueConfigurations:
            - Event: 's3:ObjectCreated:Put'
              Queue: 'arn:aws:sqs:eu-west-1:539106611526:label-generate-response'

The exact error for this resource is: 

An error occurred: CarrierLabelBucket - Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 12A3D93761EFFEAD; S3 Extended Request ID: Zfk2XKEKHhqtafaiFvrcpzyO8nHB6qOJs4gqJXpkOyhxSMgDTsUzZ0lQnYIrTEr2SVHhgMHw0ds=).

Upvotes: 4

Views: 10148

Answers (4)

Walinsky dot Com
Walinsky dot Com

Reputation: 51

Last answer by Nick is actually the correct one.

If - and when - you set up your resources S3 Bucket + SQS Queue + Policy it will work.

I did it like:

resources:
  Resources:
    MyBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${self:custom.settings.s3_bucket}
        NotificationConfiguration:
          QueueConfigurations:
            - Event: s3:ObjectCreated:Put
              Queue: "arn:aws:sqs:#{AWS::Region}:#{AWS::AccountId}:${self:custom.settings.transmit_queue}"
      DependsOn : SQSQueuePolicy

    TransmitQueue:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: ${self:custom.settings.transmit_queue}

    SQSQueuePolicy:
      Type: AWS::SQS::QueuePolicy
      Properties:
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Principal:
                AWS: "*"
              Action: SQS:SendMessage
              Resource: "*"
              Condition:
                ArnLike:
                  aws:SourceArn: "arn:aws:s3:::${self:custom.settings.s3_bucket}"
        Queues:
          - Ref: TransmitQueue

Finding this out might take you some time. Ask me how I know.

Upvotes: 5

Nick
Nick

Reputation: 77

You need to add a SQS policy to your queue before you can add the S3 SQS event.

Cloudformation SQS Policy for S3 events

Upvotes: 1

Driss Amri
Driss Amri

Reputation: 1805

I have followed the instructions on the AWS docs to create the SNS topic first in a different deployment. You can find my working application config here:

1. SNS deploy

https://github.com/drissamri/serverless-architecture/blob/master/infrastructure/serverless.yml

2. App deploy (S3 notifications)

https://github.com/drissamri/serverless-architecture/blob/master/application/serverless.yml

If you are using Serverless Framework you can also use plugins that hide all the necessary configuration with a simplified config like https://www.npmjs.com/package/@agiledigital/serverless-sns-sqs-lambda

Upvotes: -1

cementblocks
cementblocks

Reputation: 4596

You need to use s3:CreatedObject:* See https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#notification-how-to-event-types-and-destinations

Upvotes: -1

Related Questions