Why AWS Lambda function invoked multiple times for a single event?

Question

I am trying to create AWS Lambda function that does following process.

• Receive S3 "Put" event
• Get fileA from S3
• Get fileB from S3 that invoked lambda
• Launch just one EC2 instance
• Create tags for the new EC2 instance

Problem: Multiple(5) instances are launched unexpectedly.

An instance is successfully created, but 4 other instances are also launched. 5 instances in total are launched.

Logs

In the Log Streams for this function, I found 4 Streams for this invocation. Each Stream doesn't show any errors or exceptions, but it seems that the function is executed repeatedly.

Trial

I guessed that the function has been timed out and then re-run.

Then, I changed Timeout from 5s to 60s and put a file on S3. It somehow effected. Only 2 Log Streams appeared, first one shows that the function has been executed just once, second shows the function has been executed twice. Number of launched instances is 3.

However, I have no idea why multiple(3) instances are launched.

Any comments are welcome! Thank you in advance :-)

My Lambda function

My Lambda function is following. (It's simplified to hide credential informations but it doesn't lose its basic structure)

var AWS = require('aws-sdk');

function composeParams(data, config){
  var block_device_name = "/dev/xvdb";
  var security_groups = [
    "MyGroupName"
  ];
  var key_name = 'mykey';
  var security_group_ids = [
    "sg-xxxxxxx"
  ];
  var subnet_id = "subnet-xxxxxxx";

  // Configurations for a new EC2 instance
  var params = {
    ImageId: 'ami-22d27b22',      /* required */
    MaxCount: 1,                  /* required */
    MinCount: 1,                  /* required */
    KeyName: key_name,
    SecurityGroupIds: security_group_ids,
    InstanceType: data.instance_type,
    BlockDeviceMappings: [
      {
        DeviceName: block_device_name,
        Ebs: {
          DeleteOnTermination: true,
          Encrypted: true,
          VolumeSize: data.volume_size,
          VolumeType: 'gp2'
        }
      }
    ],
    Monitoring: {
      Enabled: false              /* required */
    },
    SubnetId: subnet_id,
    UserData: new Buffer(config).toString('base64'),
    DisableApiTermination: false,
    InstanceInitiatedShutdownBehavior: 'stop',
    DryRun: data.dry_run,
    EbsOptimized: false
  };

  return params;
}

exports.handler = function(event, context) {
  // Get the object from the event
  var s3 = new AWS.S3({ apiVersion: '2006-03-01' });
  var bucket = event.Records[0].s3.bucket.name;
  var key = event.Records[0].s3.object.key;

  // Get fileA
  var paramsA = {
    Bucket: bucket,
    Key: key
  };
  s3.getObject(paramsA, function(err, data) {
    if (err) {
      console.log(err);
    } else {
      var dataA = JSON.parse(String(data.Body));

      // Get fileB
      var paramsB = {
        Bucket: bucket,
        Key: 'config/config.yml'
      };
      s3.getObject(paramsB, function(err, data) {
        if (err) {
          console.log(err, err.stack);
        } else {
          var config = data.Body;
          /* Some process */

          // Launch EC2 Instance
          var ec2 = new AWS.EC2({ region: REGION, apiVersion: '2015-04-15' });
          var params = composeParams(dataA, config);
          ec2.runInstances(params, function(err, data) {
            if (err) {
              console.log(err, err.stack);
            } else {
              console.log(data);

              // Create tags for instance
              for (var i=0; i<data.Instances.length; i++){
                var instance = data.Instances[i];
                var params = {
                  Resources: [                /* required */
                    instance.InstanceId
                  ],
                  Tags: [                     /* required */
                    {
                      Key: 'Name',
                      Value: instance_id
                    },
                    {
                      Key: 'userID',
                      Value: dataA.user_id
                    }
                  ],
                  DryRun: dataA.dry_run
                };
                ec2.createTags(params, function(err, data) {
                  if (err) {
                    console.log(err, err.stack);
                  } else {
                    console.log("Tags created.");
                    console.log(data);
                  }
                });
              }
            }
          });
        }
      });
    }
  });
};

Answer 1

Maximum Event Age

When a function returns an error before execution, Lambda returns the event to the queue and attempts to run the function again for up to 6 hours by default. With Maximum Event Age, you can configure the lifetime of an event in the queue from 60 seconds to 6 hours. This allows you to remove any unwanted events based on the event age.

Maximum Retry Attempts

When a function returns an error after execution, Lambda attempts to run it two more times by default. With Maximum Retry Attempts, you can customize the maximum number of retries from 0 to 2. This gives you the option to continue processing new events with fewer or no retries.

under the Configuration > Asynchronous invocation > Retry Attempts you can set it to 0-2

Source: https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-max-retry-attempts-event-age-asynchronous-invocations/

Answer 2

Check in cloudwatch event that context.aws_request_id value for each invokation. If it is

1. same than it is retry because aws function got some error raised. make your lambda idempotent
2. different than it is because of connection timeout from your aws
   lambda client. check aws client configuration request timeout and
   connect timeout values.

Answer 3

I was having the same problem with the newer runtime (Node.JS v4.3). Call

context.callbackWaitsForEmptyEventLoop = false;

before calling

callback(...)

Answer 4

Solved.

Adding context.succeed(message); to the last part of the nested callback prevents the repeated execution of the function.

            ec2.createTags(params, function(err, data) {
              if (err) {
                console.log(err, err.stack);
                context.fail('Failed');
              } else {
                console.log("Tags created.");
                console.log(data);
                context.succeed('Completed');
              }
            });