CODEWITHSUNDEEP
amazon-web-servicesamazon-ecsenvironment
Will
Will

Reputation: 503

Update Environment Variable Secrets in an ECS Container

I am deploying a task definition for ECS that contains secrets which are passed to a running container as environment variables.

These secrets are stored in AWS Secrets Manager, and the container successfully Describes theses secrets with the correct IAM policies needed. Cool.

However, say if I were to update my secret in the Secret Manager, and change a password or something - what should I do in ECS to ensure that container gets this updated secret value?

I know that you do not need to register a new task definition, as the state has not changed - I am really looking to reboot my container.. Maybe I don't need to do anything at all?

Upvotes: 10

Views: 8108

Answers (2)

lchristmann
lchristmann

Reputation: 615

You're retrieving your AWS Secrets Manager secrets by referencing them inside your ECS task definition (docs).

Now you wonder, when you update your secrets, how do you update your ECS containers, how do they get the new secrets values?

Do you really need to to reboot them?

Yes, you really have to reboot your containers.

AWS provides an answer to that in the documentation: ECS Documentation > Task definitions > Task definition use cases > Passing sensitive data to a container > Retrieving secrets through environment variables > Using Secrets Manager (the first bullet point under Considerations)

  • Sensitive data is injected into your container when the container is initially started.
  • If the secret is subsequently updated or rotated, the container doesn't receive the updated value automatically.
  • You must either launch a new task or (if your task is part of a service) you can update the service and use the Force new deployment option to force the service to launch a fresh task.

To avoid this manual work, you can...

a) automate it - by creating a CloudWatch Event rule, that listens to changes on that secret and triggers a Lambda function, which forces a new deployment of the ECS tasks. [RECOMMENDED]

Solution:

  1. Secrets Manager secret gets updated
  2. CloudWatch Event is emitted: "Secret 'XYZ' has been changed!"
  3. TODO CloudWatch Event rule fires on the event and invokes Lambda function
  4. TODO Lambda function makes ECS API call to force new deployment for a given service
  5. Containers associated with this ECS service are stopped and restarted. As the containers restart, they receive the new secret.

Prerequisite: TODO You have to use CloudTrail (because Secrets Manager doesn't emit events itself). The whole solution is explained in this guide:

Full Guide: Updating Container Secrets Using CloudWatch Events + Lambda

Why do I recommend this solution?
->It keeps your "no-code" solution to secrets management, using ECS secrets injection via the task definition file, holding your codebase neat and small.

b) retrieve the secret dynamically - by requesting it in your application code, every time it is needed.
In order for you not to make millions of requests to the SecretsManager API (high cost, low performance!), AWS recommends to use client-side caching and has it in all of its code samples.

Java example:

public class SampleClass implements RequestHandler<String, String> {

    private final SecretCache cache  = new SecretCache();

    @Override public String handleRequest(String secretId,  Context context) {
        final String secret  = cache.getSecretString(secretId);
        // Use the secret, return success;
    }
}

Receive secrets from AWS Secrets Manager - Documentation

If you are dealing with rotating secrets:

The same two options in the section above apply.

If the secrets are in another AWS account:

You will need to implement CrossAccountEventDelivery, so you get the "Secret changed!" event in your account as well, to trigger your Lambda.

This event delivery can be done as specific as only forwarding events that your specific secret(s) changed.

Upvotes: 6

akskap
akskap

Reputation: 823

Secrets in ECS task definitions are only injected when the container starts. For this to work you need to have an ECS agent version >=1.22.0 on the container instance.

So, in the scenario mentioned above you can use one of the following options:

  • Force a new deployment (This will result in secrets being fetched again)
  • Register a new task definition (Logically the state of the definition has infact changed, because the dependent parameters/secrets were modified - it's only that you are referring an external store which is not tightly coupled with the application code)

Upvotes: 3

Related Questions