Configure AWS Lambda function to use latest version of a Layer

Question

I have more than 20 lambda functions in a developing application. And a lambda layer that contains a good amount of common code.

A Lambda function, is hook it to a particular version of the layer, and every time I update a layer, it generates a new version. Since it is a developing application, I have a new version of the layer almost every day. That creates a mess on the lambda functions that have to be touched every day - to upgrade the layer version.

I know it is important to freeze code for a lambda function in production, and it is essential to hook one version of the lambda function to a version of the layer.

But, for the development environment, is it possible to prevent generating a new layer version every time a layer is updated? Or configure the lambda function so that the latest lambda version always refers to the latest layer version?

Answer 1

From Terraform, it is possible to derive the most recent version number of a layer, using an additional data statement, as per https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/lambda_layer_version

So in your definition module, you will have the original layer resource definition

resource "aws_lambda_layer_version" "layer_mylib" {
  filename   = "layer_mylib.zip"
  layer_name = "layer_mylib"

  compatible_runtimes = ["python3.6", "python3.7", "python3.8"]
}

and then to obtain the ARN with latest version, use

data "aws_lambda_layer_version" "mylatest" {
  layer_name = aws_lambda_layer_version.layer_mylib.layer_name
}

then data.aws_lambda_layer_version.mylatest.arn

will give the reference which includes the latest version number, which can be checked by placing

output {
  value = data.aws_lambda_layer_version.mylatest.arn 
}

in your common.tf

Answer 2

Enhance from @Chris answer, you can also use a lambda-backed Custom Resource in your stack and use this lambda to update the target configuration with the new layer ARN. I note this out in case if there someone have the similar need when I found out this thread couple days ago.

There are some notes on this solution:

• The lambda of the customer resource has to send status response back to the trigger CloudFormation (CFN) endpoint, or else the CFN stack will hanging till timeout (about an hour or more, it's a painful process if you have problem on this lambda, be careful with that)
• Easy way to send response back, you can use cfnresponse (pythonic way), this lib is available magically when you use CFN lambda inline code (CFN setup this lib when processing CFN with inline code) and must have a line 'import cfnresponse' :D
• CFN will not touch to the custom resource after it created, so when you update stack for new layer change, the lambda will not trigger. A trick to make it move is to use custom resource with custom property then you will change this property with something will change each time you execute the stack, layer version arn. So this custom resource will be updated, means the lambda of this resource will be triggered when the stack update.
• Not sure why the Logical Name of the lambda layer is changed with AWS::Serverless:Layer so I can't DependOns that layer logical name but I still have !Ref its ARN

Here is a sample code

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  myshared-libraries layer

Resources:
  LambdaLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
        LayerName: !Sub MyLambdaLayer
        Description: Shared library layer
        ContentUri: my_layer/layerlib.zip
        CompatibleRuntimes: 
          - python3.7
  ConsumerUpdaterLambda:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: consumer-updater
      InlineCode: |
        import os, boto3, json
        import cfnresponse        
        def handler(event, context):
            print('EVENT:[{}]'.format(event))
            if event['RequestType'].upper() == 'UPDATE':
                shared_layer = os.getenv("DB_LAYER")
                lambda_client = boto3.client('lambda')
                consumer_lambda_list = ["target_lamda"]
                for consumer in consumer_lambda_list:
                  try:
                      lambda_name = consumer.split(':')[-1]
                      lambda_client.update_function_configuration(FunctionName=consumer, Layers=[shared_layer])
                      print("Updated Lambda function: '{0}' with new layer: {1}".format(lambda_name, shared_layer))
                  except Exception as e:
                      print("Lambda function: '{0}' has exception: {1}".format(lambda_name, str(e)))
            responseValue = 120
            responseData = {}
            responseData['Data'] = responseValue
            cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData)
      Handler: index.handler
      Runtime: python3.7
      Role: !GetAtt ConsumerUpdaterRole.Arn
      Environment:
        Variables:
          DB_LAYER: !Ref LambdaLayer

  ConsumerUpdaterRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action: sts:AssumeRole
      ManagedPolicyArns:
      - Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
      - PolicyName:
          Fn::Sub: updater-lambda-configuration-policy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - lambda:GetFunction
            - lambda:GetFunctionConfiguration
            - lambda:UpdateFunctionConfiguration
            - lambda:GetLayerVersion
            - logs:DescribeLogGroups
            - logs:CreateLogGroup
            Resource: "*"

  ConsumerUpdaterMacro:
    DependsOn: ConsumerUpdaterLambda
    Type: Custom::ConsumerUpdater
    Properties:
      ServiceToken: !GetAtt ConsumerUpdaterLambda.Arn
      DBLayer: !Ref LambdaLayer

Outputs:
  SharedLayer:
    Value: !Ref LambdaLayer
    Export:
      Name: MySharedLayer

Another option is using stack Notification ARN which send all stack events into a defined SNS, where you will use it to trigger your update lambda. In your lambda, you will filter the SNS message body (which is a readable json liked format string) with the AWS::Lambda::Layer resource then grab the PhysicalResourceId for the layer ARN. How to engage the SNS topic to your stack, use CLI sam/cloudformation deploy --notification-arns option. Unfortunately, CodePipeline doesn't support this configuration option so you can only use with CLI only

Sample code for your lambda to extract/filter the SNS message body with resource data

import os, boto3, json

def handler(event, context):
    print('EVENT:[{}]'.format(event))
    resource_data = extract_subscription_msg(event['Records'][0]['Sns']['Message'])
    layer_arn = ''
    if len(resource_data) > 0:
        if resource_data['ResourceStatus'] == 'CREATE_COMPLETE' and resource_data['ResourceType'] == 'AWS::Lambda::LayerVersion':
            layer_arn = resource_data['PhysicalResourceId']
    if layer_arn != '':
        lambda_client = boto3.client('lambda')
        consumer_lambda_list = ["target_lambda"]
        for consumer in consumer_lambda_list:
            lambda_name = consumer.split(':')[-1]
            try:
                lambda_client.update_function_configuration(FunctionName=consumer, Layers=[layer_arn])
                print("Update Lambda: '{0}' to layer: {1}".format(lambda_name, layer_arn))
            except Exception as e:
                print("Lambda function: '{0}' has exception: {1}".format(lambda_name, str(e)))
    return

def extract_subscription_msg(msg_body):
    result = {}
    if msg_body != '':
        attributes = msg_body.split('\n')
        for attr in attributes:
            if attr != '':
                items = attr.split('=')
                if items[0] in ['PhysicalResourceId', 'ResourceStatus', 'ResourceType']:
                    result[items[0]] = items[1].replace('\'', '')
    return result

Answer 3

Unfortunately it is currently not possible to reference the latest, and there is no concept of aliases for the layer versions.

The best suggestion would be to automate this, so that whenever you create a new Lambda Layer version it would update all Lambda functions that currently include this Lambda Layer.

To create this event trigger, create a CloudWatch function that uses its event to listen for the PublishLayerVersion event.

Then have it trigger a Lambda that would trigger the update-function-layers function for each Lambda to replace its layer with the new one.