CODEWITHSUNDEEP
node.jsamazon-web-servicesmessage-queueamazon-sqs
user1459144
user1459144

Reputation: 2977

AWS multiple SQS queues and workers optimal design

I have following task to implement using AWS stack:

One job is triggered periodically and put message to queue (SQS). Worker recieves this task and based on it additional tasks need to be created (approximately 1-10 K tasks). And all these tasks are also put to another queue and there are additional workers to process these tasks.
These flow can be described displayed in following way:

Periodic task ->SQS->woker_1(creates more tasks) -> SQS -> workers_2

Based on project conventions and bureaucracy it will take some time to create two separate services for worker_1 that listen to periodic task and creates fine grained tasks and for workers_2 that just process particular tasks, make docker images, CI jobs etc... and get deploy it. So, here is the tradeof:
1. Spend additional time and create two separate services. On the other hand these services might be really simple. And even there is a doubt to have 2 separate projects. 2. Make this as a one service that put messages to the same queue and also will listen to the messages on the same queue and perorm work for: worker_1 and worker_2.

Any suggestions or thoughts are appreciated!

Upvotes: 1

Views: 2412

Answers (1)

James
James

Reputation: 11931

I don't think there can be a "correct" answer to this, you already have a good list of pros and cons for both options. Some additional things I thought of:

  • SQS queues don't really allow you to pick out specific types of messages, you pretty much need to read everything first-in-first-out. So if you share queues, you may have less control of prioritizing messages.
  • For the two services to interact, they need a shared message definition. Sharing the same codebase would make it easier to dev and test the messaging code. Of course, it could also be a shared library.
  • Deploying both worker types in the same server/application would share resources, which might be more economical at the low end, or it might be confusing at high scale.
  • It may be possible to develop all the code into the same application, and leave the decision to deployment-time if it is all on the same server and queue or separate servers reading from separate queues. This seems ideal to me.

Upvotes: 3

Related Questions