How can I update and ECS service adding an addition load balance to which a service talks to without downtime?

Question

We use terraform to manage our AWS resources and have the code to change the service from one to two load balancers.

However, terraform wants to destroy the service prior to recreating it. AWS cli docs indicate the reason - the API can only modify LBs during service creation and not on update.

• create-service
• update-service

It seems we need a blue/green deploy with the one LB and two LB services existing at the same time on the same cluster. I expect we'll need to create multiple task sets and the rest of the blue/green approach prior to this change (we'd planned for this anyway just not at this moment)

Does anyone have a good example for this scenario or know of any other approaches beyond full blue/green deployment?

Answer 1

As of June 2024 it's possible to update (add/remove) load balancers for ECS service using simple JS script. I have the same problem and AWS CLI doesn't have option to update load balancers for existing service but you can use AWS SDK for JS and AWS CloudShell to run JS script.

I created simple script which updates load balancers for given ECS service. Before running it you must create target group and assign it to load balancer. After that copy ARN of target group(s) you like to assign service to into below script.

Before running script, install AWS SDK for JS version 3 using: npm i @aws-sdk/client-ecs

In below code I marked using TODO places where you need to put container details, cluster name and target groups:

import { ECSClient, UpdateServiceCommand } from "@aws-sdk/client-ecs";
const client = new ECSClient();

const containerName = 'YOUR CONTAINER NAME' // TODO
const containerPort = 80 // TODO: change it if needed

const input = {
  service: 'YOUR SERVICE NAME', // TODO
  cluster: 'YOUR_CLUSTER_NAME', // TODO
  loadBalancers: [
    {
      "targetGroupArn": "arn:aws:elasticloadbalancing:eu-west-1:1234567:targetgroup/target-group-1/123456", // TODO: first target group
      "containerName": containerName,
      "containerPort": containerPort
    },
    {
      "targetGroupArn": "arn:aws:elasticloadbalancing:eu-west-1:1234567:targetgroup/target-group-2/123456", // TODO: second target group
      "containerName": containerName,
      "containerPort": containerPort
    }
  ]
}

const command = new UpdateServiceCommand(input);
const response = await client.send(command);

console.log(response)

IMPORTANT: Please save it with .mjs (ES module) not .js extension! It's required because in script I use ES modules import and await.

If everything goes ok, you will see response with details of your service with updated configuration. Otherwise, you will error with details.

Answer 2

I ran into the same problem recently so I felt I should share my knowledge.

The info provided in the other answers are right, what you would need is the --load-balancers parameter in the AWS CLI. Unfortunately, that parameter is only available in the v1 of the CLI, it got removed in the v2.

I solved that by installing the old CLI, using the instructions here: https://docs.aws.amazon.com/cli/v1/userguide/cli-chap-install.html

Note that I had to use python 3.8 to install it, the installer wasn't working with python 3.12. On my linux setup it installed the CLI in a local folder, so I was able to use both v1 and v2 side by side, which was very handy!

Once installed, you just have to run the following:

/path/to/v1/aws ecs update-service --cluster <cluster-name> --service <service-name> --load-balancers '[
  {
    "targetGroupArn": "<target-group-arn>",
    "containerName": "<container-name>",
    "containerPort": <container-port>
  },
  {
    "targetGroupArn": "<target-group-arn>",
    "containerName": "<container-name>",
    "containerPort": <container-port>
  }
]'

Answer 3

To update an Amazon ECS service with multiple load balancers, you need to ensure that you are using version 2 of the AWS CLI. Then, you can run the following command:

aws ecs update-service \
    --cluster <cluster-name> \
    --service <service-name> \
    --load-balancers "[{\"containerName\": \"<container-name>\", \"containerPort\": <container-port>, \"targetGroupArn\": \"<target-group-arn1>\"}, {\"containerName\": \"<container-name>\", \"containerPort\": <container-port>, \"targetGroupArn\": \"<target-group-arn2>\"}]"

In the above command, replace with the name of your ECS cluster, with the name of your ECS service, with the name of the container in your task definition, with the port number the container listens to, and and with the ARN of your target groups.

Answer 4

Update 2022

ECS now supports updation of load balancers of a service. For now, this can be achieved using AWS CLI, AWS SDK, or AWS Cloudformation (not from AWS Console). From the documentation:

When you add, update, or remove a load balancer configuration, Amazon ECS starts new tasks with the updated Elastic Load Balancing configuration, and then stops the old tasks when the new tasks are running.

So, you don't need to create a new service. AWS update regarding this here. Please refer Update Service API doc on how to make the request.

Answer 5

Alas, it is not possible to change the number of LBs during an update. The service must be destroyed and recreated.

Ideally, one would be doing blue green deploys with multiple ECS clusters and a set of LBs. Then cluster A could have the old service and cluster B have the new service allowing traffic to move from A to B as we go from blue to green.

We aren't quite there yet but plan to be soon. So, for now, we will do the classic parking lot switch approach:

In this example the service that needs to go from 1 LB to 2 LBs is called target_service

1. clone target_service to become target_service2
2. deploy microservices that can talk to either target_service or target_service2
3. verify that target_service and target_service2 are both handling incoming data and requests
4. modify target_service infra-as-code to go from 1 to 2 LBs
5. deploy modified target_service (terraform deployment tool will destroy target_service leaving target_service2 to cover the gap, and then it will deploy target_service with 2 LBs
6. verify that target_service with 2 LBS works and handle requests
7. destroy and remove target_service2 as it is no longer needed

So, this is a blue-green like deploy albeit less elegant.