CODEWITHSUNDEEP
aws-lambdacontainersamazon-cloudwatchslack-apiwinston
Nenad
Nenad

Reputation: 271

Using cloudwatch and slack transports for winston under Lambda environment

im running a service under Lambda containers as its easier for me to have a way to easily run it as standalone server if needed because its the same Docker image. For lambda I include public.ecr.aws/awsguru/aws-lambda-adapter:0.8.4 and it gives me what I need. My service uses Winston to log stuff and I wrote Slack transport for errors (using slack webhooks) and Im using https://www.npmjs.com/package/winston-cloudwatch for cloudwatch logs. Things I want to understand are:

What I think the problem is is that lambda as soon as request is finished freezes code so this async request cannot be completed, but why does it happend when container shuts down, I can share my implementation and would appreciate help with this. Thanks!

Upvotes: 0

Views: 96

Answers (1)

fa44
fa44

Reputation: 502

on first invoke when lambda pulls new image or creates container, if result of first invoke is error (if i intentionally send bad request params) slack transport will send error only when that container is killed (aprox. 5mins after there are no new requests), if i try new requests slack transport will send them to slack

This looks like a lambda leak issue. It happens in your case because Lambda is getting resumed just before the shutdown (Note that this part is not documented anywhere but I suppose that the freeze happens with a SIGSTOP signal, and SIGCONT signal is called before SIGTERM)

To avoid those type of issues, you should:

  • Write your asynchronous code using promises if you have asynchronous handler (check an example below)
  • Make sure your last callback is calling the main callback handler with success/error status if you're using callback based handler.

winston cloudwatch transport will not log any error to cloudwatch during container lifetime but will send them all after container is killed when lambda decide to do that

Like previous one, this issue seems to be related to asynchronous handling issue. There are already some issues regarding that subject opened on the winston-cloudwatch library. You can check this answer and this working example. The idea here is to promisify the callback function to use it in an asynchronous Lambda handler:

const winstonCloudWatch = new WinstonCloudWatch({
    name: 'using-kthxbye',
    logGroupName: 'testing',
    logStreamName: 'another',
    awsRegion: 'us-east-1',
    awsAccessKeyId: await getSecretValue('AWS_ACCESS_KEY_ID'),
    awsSecretKey: await getSecretValue('AWS_SECRET_KEY'),
});
kthxbyeAsync = promisify(winstonCloudWatch.kthxbye).bind(winstonCloudWatch);
winston.add(winstonCloudWatch);

winston.add(new winston.transports.Console({
    level: 'info'
}));

Upvotes: 0

Related Questions