CODEWITHSUNDEEP
amazon-web-servicesamazon-sqs
bfallik
bfallik

Reputation: 2995

sharing SQS across multiple environments

We're exploring SQS to improve the reliability of some asynchronous job queues. I'm trying to identify the best SQS deployment strategy to support multiple queues and environments (farm clusters, developer sandboxes, developer laptops, etc.). Our previous job queue used a separate queue server for each environment which provided for nice isolation. Since SQS is a global resource I don't quite yet see the optimal path for security/isolation and maintenance.

In the worse case I think we'd need to create thousands of individual SQS queues to handle each combination of job type (e.g. reports, ETL, etc.) and environment (production, demo, developer sandboxes, etc.). We could use a single queue per environment but that seems to limit our ability to control behavior based on the job. Conversely we could use global queues for each job but that makes it impossible to isolate the environments.

This seems like a bit of a maintenance and security burden so I'm wondering how other customers have successfully leverages SQS across a large set of environments and queue payload types.

Upvotes: 3

Views: 2239

Answers (2)

JaredHatfield
JaredHatfield

Reputation: 6671

While using completely separate accounts is an appropriate best practice as described in Mark B's answer, it is not the only one. You can have multiple environments in the same AWS account and achieve isolation, just at a different level.

To isolate different environments you need to apply naming standards to your resources, you can then use IAM to restrict certain access keys and roles to only have permission to the resources (ex: SQS queues) that are associated with that environment.

For example, you could have the following queues:

  • etl-development
  • etl-demo
  • etl-production
  • report-development
  • report-demo
  • report-production

You can then apply least privileged to the roles that are access these queues to only allow the corresponding applications per environment to have access to them. If you are using CloudFormation it is possible to specify the environment portion of the resource name as a parameter to the stack.

That said, isolating at the account level provides stronger guarantees. Depending on your acceptable level of risk and compliance policies you may need to comply with separate accounts provides some additional protections. A hybrid approach is also possible, such as having a separate account just for sandbox development, but have your pre-production and production resources all in one account. If you do end up using a single account naming conventions are really the key and a good dose of automation with tools such as Jenkins and CloudFormation can help with that.

Upvotes: 3

Mark B
Mark B

Reputation: 200998

I would split each environment (prod/demo/etc.) into separate AWS accounts to provide isolation of environments.

I would recommend using something like CloudFormation or Teraform to create and manage the all the queues (and other AWS resources), which should reduce your maintenance and security burden.

Upvotes: 3

Related Questions