CODEWITHSUNDEEP
javaamazon-ecsscheduledexecutorservice
Almas Abdrazak
Almas Abdrazak

Reputation: 3622

Start scheduled job inside of a Java app deployed in ECS fargate after previous task will be shutdown

Good day everyone. I have a problem with my java app deployed in AWS ECS Fargate instance. With our configuration the fargate instance is running all the time and during the redeploy , ECS spawn a new instance(instance A) waits for it to reach steady state and then shuts down the old instance (instance B) The problem is that the main method of these instances start a scheduled job. And because instance A reaches the steady state , the scheduled job is executed by both instances for a short period of time until B will be shut down. How to make sure instance A starts a job only when B doesn't work ? The java instance is dockerized So far I tried

  1. Runtime.addShutdownHook - doesn't work because proprietary java framework that we use require us to boot it up with a shell script (PID 1 is given to shell rather than java app)
  2. Self written cluster healthcheck - self written code that sends heartbeat messages from the instance and saves them in Dynamo, once instance B doesn't work , instance A checks that last hearbeat timestamp and if it's longer than 30 seconds then starts a job
  3. Event bridge - ecs docs says that it publishes events to event bridge. But my customer doesn't want to pay for another service

All suggestions will be appreciated !

Upvotes: 2

Views: 845

Answers (1)

Mark B
Mark B

Reputation: 200790

  1. Runtime.addShutdownHook - doesn't work because proprietary java framework that we use require us to boot it up with a shell script (PID 1 is given to shell rather than java app)

You could capture SIGTERM in your shell script, and trigger a shutdown of the Java app from that. However I don't think that happens until right before the container is killed, which isn't going to help anything in your situation. All the Java shutdown hook would do is allow you to close some things out right before your container is killed, it wouldn't trigger until the very end of your deployment process, where it seems like you need it to run as soon as the deployment starts.

  1. Self written cluster healthcheck - self written code that sends heartbeat messages from the instance and saves them in Dynamo, once instance B doesn't work , instance A checks that last hearbeat timestamp and if it's longer than 30 seconds then starts a job

If you were using Quartz scheduler you could use a database backend for the scheduler and it would handle this sort of concurrency thing automatically. It sounds like you are using some sort of proprietary in-house scheduler solution, so if you want a database-backed job concurrency solution like this you will have to build it yourself, and DynamoDB is a good option.

  1. Event bridge - ecs docs says that it publishes events to event bridge. But my customer doesn't want to pay for another service

AWS Free Tier: "All state change events published by AWS services by default are free."

There is no additional cost for using EventBridge if you are only using it for AWS service events. You are literally already using EventBridge in the sense that AWS is already publishing all those ECS service events to EventBridge. You just aren't listening to them currently.

Upvotes: 1

Related Questions