How to launch ECS task instances dynamically using C#?

Question

I have a micro-services application that I am deploying to AWS using ECS Fargate. Most of the containers can be deployed as long running tasks in a service. However, I need to spin up new containers as and when I receive JSON messages over a ZMQ REQ/REP port. These containers need to be spin up and need to run as long running ECS tasks.

How do I dynamically launch such AWS Tasks? How does the Service monitor such tasks?

I tried researching the internet and people are mentioning AWS Batch but my use-case is a bit different as I want this dynamically launched task to be a long running task similar to the tasks in an ECS service.

Answer 1

then you to use following nuget pkg in your some persistent service like EC2 or some log running ECS task itself

dotnet add package AWSSDK.ECS
dotnet add package NetMQ
dotnet add package Newtonsoft.Json

then

you can write this type of code listener

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading.Tasks;
using Amazon;
using Amazon.ECS;
using Amazon.ECS.Model;
using NetMQ;
using NetMQ.Sockets;
using Newtonsoft.Json;

class Program
{
    // AWS ECS Configuration
    private static readonly string ClusterName = "my-cluster"; // Update with your ECS cluster name
    private static readonly string TaskDefinition = "my-task-definition"; // Update with your ECS task definition
    private static readonly List<string> Subnets = new() { "subnet-123456" }; // Update with your VPC subnets
    private static readonly List<string> SecurityGroups = new() { "sg-123456" }; // Update with your security groups
    private static readonly AmazonECSClient EcsClient = new AmazonECSClient(RegionEndpoint.USEast1); // Update with your AWS region

    static async Task Main(string[] args)
    {
        Console.WriteLine("Starting ZMQ Listener...");

        using (var responder = new ResponseSocket("@tcp://*:5555")) // ZMQ Listener on Port 5555
        {
            while (true)
            {
                string request = responder.ReceiveFrameString();
                Console.WriteLine($"Received ZMQ Message: {request}");

                try
                {
                    var payload = JsonConvert.DeserializeObject<Dictionary<string, string>>(request);

                    if (payload != null && payload.ContainsKey("taskArgs"))
                    {
                        string taskArgs = payload["taskArgs"];
                        string taskArn = await RunEcsTaskAsync(taskArgs);

                        responder.SendFrame($"Task started: {taskArn}");
                    }
                    else
                    {
                        responder.SendFrame("Invalid request: Missing taskArgs");
                    }
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"Error processing request: {ex.Message}");
                    responder.SendFrame("Error processing request");
                }
            }
        }
    }

    /// <summary>
    /// Runs an ECS Fargate task dynamically with provided arguments.
    /// </summary>
    private static async Task<string> RunEcsTaskAsync(string taskArgs)
    {
        var request = new RunTaskRequest
        {
            Cluster = ClusterName,
            LaunchType = LaunchType.FARGATE,
            TaskDefinition = TaskDefinition,
            Count = 1,
            Overrides = new TaskOverride
            {
                ContainerOverrides = new List<ContainerOverride>
                {
                    new ContainerOverride
                    {
                        Name = "my-container", // Update with your container name
                        Command = new List<string> { taskArgs }
                    }
                }
            },
            NetworkConfiguration = new NetworkConfiguration
            {
                AwsvpcConfiguration = new AwsVpcConfiguration
                {
                    Subnets = Subnets,
                    SecurityGroups = SecurityGroups,
                    AssignPublicIp = AssignPublicIp.ENABLED
                }
            }
        };

        try
        {
            var response = await EcsClient.RunTaskAsync(request);

            if (response.Tasks.Count > 0)
            {
                string taskArn = response.Tasks[0].TaskArn;
                Console.WriteLine($"Task started successfully: {taskArn}");
                return taskArn;
            }
            else
            {
                Console.WriteLine("Failed to start ECS task.");
                return null;
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error launching ECS task: {ex.Message}");
            return null;
        }
    }
}

and create write role that you allow to assume your docker/EC2 code so that can excute this code