Cloudformation: Invalid permissions on Lambda function found with API GW Integration

Question

This is my cloudformation template, which deploys successfully:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Deploy Da Blog

Resources:
  ArticleTable:
    Type: AWS::DynamoDB::Table
    Properties:
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      ProvisionedThroughput:
        ReadCapacityUnits: 1
        WriteCapacityUnits: 1
      TableName: !Sub ${AWS::StackName}-Article
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain

  ############################### BLOG API ############################
  BlogRestApiLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/aws/apigateway/${AWS::StackName}'

  BlogRestApiLoggingRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub 'BlogRestApiLoggingRole-${AWS::StackName}'
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: apigateway.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs

  ## specifies the IAM role that Amazon API Gateway uses to write API logs to Amazon CloudWatch Logs
  BlogRestApiAccount:
    Type: AWS::ApiGateway::Account
    Properties:
      CloudWatchRoleArn: !GetAtt [ BlogRestApiLoggingRole, Arn ]

  BlogRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: !Sub '${AWS::StackName}'
      Description: Blog API
      EndpointConfiguration:
        Types:
          - EDGE
  BlogRestApiDeployment:
    Type: AWS::ApiGateway::Deployment
    Properties:
      RestApiId: !Ref BlogRestApi
      Description: Automatically created by the RestApi construct
    DependsOn:
      - BlogRestApi
      - BlogApiResource
      - BlogApiIdResource
      - ListArticlesMethod
      - PostArticleMethod
      - GetArticleMethod

  BlogRestApiStageProd:
    Type: AWS::ApiGateway::Stage
    DependsOn:
      - BlogRestApiDeployment
    Properties:
      RestApiId: !Ref BlogRestApi
      DeploymentId: !Ref BlogRestApiDeployment
      StageName: prod
      MethodSettings:
        - LoggingLevel: INFO
          ResourcePath: '/*'
          HttpMethod: '*'
          MetricsEnabled: true

  BlogApiRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub 'BlogApiRole-${AWS::StackName}'
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: apigateway.amazonaws.com
            Action:
              - sts:AssumeRole

  BlogApiPolicy:
    Type: AWS::IAM::Policy
    DependsOn:
      - BlogApiRole
      - ListArticlesFunction
      - PostArticleFunction
      - GetArticleFunction
    Properties:
      PolicyName: !Sub 'BlogApiPolicy-${AWS::StackName}'
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: 'Allow'
            Action: 'lambda:InvokeFunction'
            Resource:
              - !GetAtt ListArticlesFunction.Arn
              - !GetAtt PostArticleFunction.Arn
              - !GetAtt GetArticleFunction.Arn
      Roles:
        - !Ref BlogApiRole
  ############################### END OF Blog API ############################

  ############################### START OF Blog Functions ####################
  BlogApiResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !GetAtt [ BlogRestApi, RootResourceId ]
      PathPart: article
      RestApiId: !Ref BlogRestApi

  BlogApiIdResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !Ref BlogApiResource
      PathPart: "{id}"
      RestApiId: !Ref BlogRestApi

  BlogApiOptionsMethod:
    Type: AWS::ApiGateway::Method
    DependsOn:
      - BlogApiResource
    Properties:
      ApiKeyRequired: true
      AuthorizationType: NONE
      HttpMethod: OPTIONS
      Integration:
        Type: MOCK
        PassthroughBehavior: WHEN_NO_TEMPLATES
        RequestTemplates:
          'application/json': '{"statusCode": 200}'
        IntegrationResponses:
          - StatusCode: 200
            ResponseParameters:
              method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
              method.response.header.Access-Control-Allow-Methods: "'GET,POST,PATCH,OPTIONS'"
              method.response.header.Access-Control-Allow-Origin: "'*'"
            ResponseTemplates:
              'application/json': '{}'
      MethodResponses:
        - StatusCode: 200
          ResponseParameters:
            method.response.header.Access-Control-Allow-Headers: false
            method.response.header.Access-Control-Allow-Methods: false
            method.response.header.Access-Control-Allow-Origin: false
      ResourceId: !Ref BlogApiResource
      RestApiId: !Ref BlogRestApi


  ListArticlesFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub '${AWS::StackName}-list-articles-function'
      Handler: com.zenithwebfoundry.blog.api.ListArticlesHandler
      CodeUri:
        Key: !Ref ParamCodePackage
        Bucket: !Ref ParamCodeBucket
      Policies:
        - Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - dynamodb:Query
              Resource: !GetAtt [ ArticleTable, Arn ]
      Runtime: java11
      Timeout: 10
      MemorySize: 256
      Environment:
        Variables:
          TABLE_NAME: !Ref ArticleTable
          PRIMARY_KEY: id
    DependsOn:
      - ArticleTable

  ListArticlesMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: GET
      ResourceId: !Ref BlogApiResource
      RestApiId: !Ref BlogRestApi
      AuthorizationType: NONE
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS
        Uri: !Sub 'arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ListArticlesFunction.Arn}/invocations'
        PassthroughBehavior: WHEN_NO_TEMPLATES
        RequestTemplates:
          'application/json': !Ref ParamAPIGatewayRequestEventMappingTemplate
        IntegrationResponses:
          - StatusCode: 200
            ResponseTemplates:
              'application/json': !Ref ParamAPIGatewayResponseEventMappingTemplate
      MethodResponses:
        - StatusCode: 200
      OperationName: ListArticles

  PostArticleFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub '${AWS::StackName}-post-article-function'
      Handler: com.zenithwebfoundry.blog.api.SaveArticleHandler
      CodeUri:
        Key: !Ref ParamCodePackage
        Bucket: !Ref ParamCodeBucket
      Policies:
        - Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - dynamodb:PutItem
              Resource: !GetAtt [ ArticleTable, Arn ]
      Runtime: java11
      Timeout: 10
      MemorySize: 256
      Environment:
        Variables:
          TABLE_NAME: !Ref ArticleTable
          PRIMARY_KEY: id
    DependsOn:
      - ArticleTable

  PostArticleMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: POST
      ResourceId: !Ref BlogApiResource
      RestApiId: !Ref BlogRestApi
      AuthorizationType: NONE
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS
        Uri: !Sub 'arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${PostArticleFunction.Arn}/invocations'
        PassthroughBehavior: WHEN_NO_TEMPLATES
        RequestTemplates:
          'application/json': !Ref ParamAPIGatewayRequestEventMappingTemplate
        IntegrationResponses:
          - StatusCode: 200
            ResponseTemplates:
              'application/json': !Ref ParamAPIGatewayResponseEventMappingTemplate
      MethodResponses:
        - StatusCode: 200
      OperationName: PostArticle


  GetArticleFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub '${AWS::StackName}-get-article-function'
      Handler: com.zenithwebfoundry.blog.api.GetArticleHandler
      CodeUri:
        Key: !Ref ParamCodePackage
        Bucket: !Ref ParamCodeBucket
      Policies:
        - Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - dynamodb:GetItem
              Resource: !GetAtt [ ArticleTable, Arn ]
      Runtime: java11
      Timeout: 10
      MemorySize: 256
      Environment:
        Variables:
          TABLE_NAME: !Ref ArticleTable
          PRIMARY_KEY: id
    DependsOn:
      - ArticleTable

  GetArticleMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: GET
      ResourceId: !Ref BlogApiIdResource
      RestApiId: !Ref BlogRestApi
      AuthorizationType: NONE
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS
        Uri: !Sub 'arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GetArticleFunction.Arn}/invocations'
        PassthroughBehavior: WHEN_NO_TEMPLATES
        RequestTemplates:
          'application/json': !Ref ParamAPIGatewayRequestEventMappingTemplate
        IntegrationResponses:
          - StatusCode: 200
            ResponseTemplates:
              'application/json': !Ref ParamAPIGatewayResponseEventMappingTemplate
      MethodResponses:
        - StatusCode: 200
      OperationName: GetArticle

  ############################### END OF Blog Functions ######################

  BlogAPIDomainName:
    Type: AWS::ApiGateway::DomainName
    Properties:
      DomainName: !Join [ ".", ['blogapi', !Ref DomainName]]
      EndpointConfiguration:
        Types:
          - EDGE
      CertificateArn: !Ref DomainCert
      SecurityPolicy: TLS_1_0
  BlogAPIHostedZone:
    Type: AWS::Route53::HostedZone
    Properties:
      Name: !Ref BlogAPIDomainName
  BlogAPIBasePathMapping:
    Type: AWS::ApiGateway::BasePathMapping
    Properties:
      DomainName: !Ref BlogAPIDomainName
      RestApiId: !Ref BlogRestApi
      Stage: 'prod'
  Route53RecordSetGroup:
    Type: AWS::Route53::RecordSetGroup
    Properties:
      HostedZoneId: !Ref DomainHostedZoneId
      RecordSets:
        - Name: !Join [ "", ['blog', '.', !Ref DomainName, '.']]
          Type: A
          TTL: '300'
          ResourceRecords:
            - 52.64.238.177
        - Name: !Join [ ".", ['blogapi', !Ref DomainName]]
          Type: A
          AliasTarget:
            # HostedZoneId: !Ref BlogAPIHostedZone  # distributionHostedZoneId - alias target name does not lie in the target zone
            HostedZoneId: Z2FDTNDATAQYW2
            # https://j97h8bvml9.execute-api.ap-southeast-2.amazonaws.com/prod/articles
            DNSName: !GetAtt [BlogAPIDomainName, DistributionDomainName]

  AssetsBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Join [ ".", [ !Ref 'AWS::StackName', 'assets' ] ]
      CorsConfiguration:
        CorsRules:
          - AllowedHeaders: ['*']
            AllowedMethods: [GET,PUT,POST,DELETE,HEAD]
            AllowedOrigins: ['http://localhost*']
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
  WebBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Join [ ".", [ !Ref 'AWS::StackName', 'web' ] ]
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      WebsiteConfiguration:
        IndexDocument: 'index.html'
        ErrorDocument: 'index.html'



Parameters:
  ParamCodePackage:
    Type: String
  ParamCodeBucket:
    Type: String
  DomainName:
    Description: "Public DNS Zone Name"
    Type: String
  DomainHostedZoneId:
    Type: String
    Description: 'The AWS HostedZoneId of the above domain name'
  DomainCert:
    Type: String
    Description: 'The arn reference to the certificate used with the domain.'
  ParamRequestMappingTemplate:
    Type: String
    Description: 'Read from resources/templates'
  SaveArticleHandler:
    Type: String
    Default: 'com.zenithwebfoundry.blog.api.SaveArticleHandler'
  GetArticleHandler:
    Type: String
    Default: 'com.zenithwebfoundry.blog.api.GetArticleHandler'
  ListArticlesHandler:
    Type: String
    Default: 'com.zenithwebfoundry.blog.api.ListArticlesHandler'
  ParamAPIGatewayRequestEventMappingTemplate:
    Type: String
    Default: '{
                  "resource" : "$context.resourceId",
                  "path" : "$context.path",
                  "httpMethod" : "$context.httpMethod",
                  "headers": {
                    #foreach($header in $input.params().header.keySet())
                      "$header": "$util.escapeJavaScript($input.params().header.get($header))" #if($foreach.hasNext),#end
                    #end
                  },
                  "method": "$context.httpMethod",
                  "pathParameters": {
                    #foreach($param in $input.params().path.keySet())
                      "$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
                    #end
                  },
                  "queryStringParameters": {
                      #foreach($queryParam in $input.params().querystring.keySet())
                          "$queryParam": "$util.escapeJavaScript($input.params().querystring.get($queryParam))" #if($foreach.hasNext),#end
                      #end
                  },
                  "body" : $input.json("$"),
                  "isBase64Encoded": false
                }'

  ParamAPIGatewayResponseEventMappingTemplate:
    Type: String
    Default: '#set($statusCode = $input.path("$.statusCode"))
                #set($context.responseOverride.status = $statusCode)

                #set($headers = $input.path("$.headers"))
                #foreach($key in $headers.keySet())
                  #set($context.responseOverride.header[$key] = $headers[$key])
                #end
                #set($context.responseOverride.header.Access-Control-Allow-Headers = "Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token")
                #set($context.responseOverride.header.Access-Control-Allow-Methods = "*")
                #set($context.responseOverride.header.Access-Control-Allow-Origin = "*")
                {
                  "body": $input.json("$.body")
                }
                '

Outputs:
  ArticleEndpoint:
    Value: !Join ["", ['https://', !Ref BlogRestApi, '.execute-api.ap-southeast-2.', !Ref 'AWS::URLSuffix', '/', !Ref BlogRestApiStageProd, '/'] ]

The problem is that when I try to call the ListArticlesFunction using the following curl expression:

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X GET https://blogapi.zenithwebfoundry.com/article

I get the following:

HTTP/2 500 
content-type: application/json
content-length: 36
date: Sun, 30 Aug 2020 09:09:50 GMT
x-amzn-requestid: c1948ed5-18c5-4807-8904-3c1c2af06c25
x-amzn-errortype: InternalServerErrorException
x-amz-apigw-id: SE3y1G71ywMF5fw=
x-cache: Error from cloudfront
via: 1.1 647846f53eba457a8e4ba1d1d42a6336.cloudfront.net (CloudFront)
x-amz-cf-pop: SYD1-C1
x-amz-cf-id: 6bnMhNRUdf1znTmD0vQn86UZcMF_j9JCzFb-JvhwwT9j6ch4P8t20g==

{"message": "Internal server error"}

Checking cloudwatch, I see the slightly cryptic error:

Execution failed due to configuration error: Invalid permissions on Lambda function

The same thing happens if I go into the APIGateway service console and go to the article GET resource. So at least its consistent.

I have tested the handler in isolation and there's no compilation problems anywhere, so I'm pretty sure its stack-related.

I gather that the problem must either be in the BlogApiPolicy Policy or the Lambda Policy, but nothing I set seems to work. Does anyone know what cloudformation shenanigans is needed to get this right?

Answer 1

Based on the comments.

The issue was that BlogApiRole, although being created, it was not used in any of the API Gateway methods. This is required, because API Gateway needs to have explicit permissions to invoke a lambda functions. To enable this, Credentials in AWS::ApiGateway::Method Integration should be set.

The alternative is to use AWS::Lambda::Permission. The use of the AWS::Lambda::Permission is especially useful when there are many methods in the API gateway which require invoking the lambda. The reason is that you can create one such AWS::Lambda::Permission for the lambda function which can allow API gateway access to the lambda as a whole. This saves us from defining Credentials for each method independently.

Answer 2

The answer is that I needed the role added as a Credentials: entry in the method of each Lambda, for example:

  ListArticlesMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: GET
      ResourceId: !Ref BlogApiResource
      RestApiId: !Ref BlogRestApi
      AuthorizationType: NONE
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS
        Uri: !Sub 'arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ListArticlesFunction.Arn}/invocations'
        Credentials: !GetAtt [BlogApiRole, Arn]
        PassthroughBehavior: WHEN_NO_TEMPLATES
        RequestTemplates:
          'application/json': !Ref ParamAPIGatewayRequestEventMappingTemplate
        IntegrationResponses:
          - StatusCode: 200
            ResponseTemplates:
              'application/json': !Ref ParamAPIGatewayResponseEventMappingTemplate
      MethodResponses:
        - StatusCode: 200
      OperationName: ListArticles

Why this is suddenly needed, when this was never needed before, is quite beyond me, but I must thank @Marcin for his/her patient assistance and convincing my doubting mind.