CODEWITHSUNDEEP
amazon-web-servicesaws-lambdanestjsaws-secrets-manager
Konstantin Bodnia
Konstantin Bodnia

Reputation: 1523

AWS Secrets manager in Nest JS microservice (lambda) fails periodically

I'm having a lambda function made with NestJS's microservice. It uses a database connection and I'm using a secret service to fetch connection details for it.

Here's my app module:

@Module({
  imports: [
    ConfigModule,
    TypeOrmModule.forRootAsync({
      useClass: SecretsService,
      inject: [],
      imports: [ConfigModule],
    }),
    PropertyModule,
  ],
})
export class AppModule {}

And this is a Secret Service (a part of the ConfigModule):

import { Injectable } from '@nestjs/common';
import { SecretsManager } from 'aws-sdk';
import { GetSecretValueResponse } from 'aws-sdk/clients/secretsmanager';
import { MysqlConnectionOptions } from 'typeorm/driver/mysql/MysqlConnectionOptions';

@Injectable()
export class SecretsService /* 👁🔫🩸 */ {

  private secretsManager: SecretsManager;

  constructor() {
    this.secretsManager = new SecretsManager();
  }

  async createTypeOrmOptions(): Promise<MysqlConnectionOptions> {
    console.log('before getting secret');
    const { SecretString }: GetSecretValueResponse =
      await this.secretsManager.getSecretValue({ SecretId: 'rds/prod' }).promise();
    const secret = JSON.parse(SecretString);
    console.log('after getting a secret', SecretString);

    return {
      /* database config */
    };
  }
}

And it turns out that the code doesn't always get to the “after getting a secret” part. Here are some cases

I change something in the code and deploy a new version of the lambda and it just keeps hanging at the “before getting secret” forever. I wait for 5 minutes and fire that function again, then I wait 10 minutes. Same result.

Then I wait like 20 minutes and the request slips through. After that, I can fire the same function several times in a row and I see “after getting secret” every time.

So it is in fact not fails periodically, but works periodically. Seems like there's some sort of throttling and/or caching, but I don't see it in the code.

Please help me to solve this issue. How can I get my secrets every time I want them?

Upvotes: 4

Views: 2427

Answers (2)

Konstantin Bodnia
Konstantin Bodnia

Reputation: 1523

The lambda belonged to three subnets one of which was public and two were private. And in the end, it only worked with one subnet, because the rest were poorly configured by our cluster architect.

Took me ages to dig to the roots of the problem. Check carefully how your network is configured.

Upvotes: 4

jarmod
jarmod

Reputation: 78693

You should use client-side caching and backoff/retry when accessing Secrets Manager from AWS Lambda.

For more, see Secrets Manager Best Practices.

Upvotes: 2

Related Questions