CODEWITHSUNDEEP
node.jsscheduled-tasksamazon-elb
efreezy
efreezy

Reputation: 273

Task scheduling behind multiple instances

Currently I am solving an engineering problem, and want to open the conversation to the SO community.

I want to implement a task scheduler. I have two separate instances of a nodeJS application sitting behind an elastic load balancer (ELB). The problem is when both instances come up, they try to execute the same tasks logic, causing the tasks run more than once.

My current solution is to use node-schedule to schedule tasks to run, then have them referencing the database to check if the task hasn't already been run since it's specified run time interval.

The logic here is a little messy, and I am wondering if there is a more elegant way I could go about doing this.

Perhaps it is possible to set a particular env variable on a specific instance - so that only that instance will run the tasks.

What do you all think?

Upvotes: 1

Views: 1432

Answers (1)

Xavier Hutchinson
Xavier Hutchinson

Reputation: 2227

What you are describing appears to be a perfect example of a use case for AWS Simple Queue Service.

https://aws.amazon.com/sqs/details/

Key points to look out for in your solution:

  • Make sure that you pick a visibility timeout that is reflective of your workload (so messages don't reenter the queue whilst still in process by another worker)
  • Don't store your workload in the message, reference it! A message can only be up to 256kb in size and message sizes have an impact on performance and cost.
  • Make sure you understand billing! As billing is charged in 64KB chunks, meaning 1 220KB message is charged as 4x 64KB chucks / requests.
  • If you make your messages small, you can save more money by doing batch requests as your bang for buck will be far greater!
  • Use longpolling to retrieve messages to get the most value out of your message requests.
  • Grant your application permissions to SQS by the use of an EC2 IAM Role, as this is the best security practice and the recommended approach by AWS.

It's an excellent service, and should resolve your current need nicely.

Thanks!

Xavier.

Upvotes: 1

Related Questions