CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2apache-stormmesosmesosphere
Bhargav Sarvepalli
Bhargav Sarvepalli

Reputation: 300

Strom on Mesos in AWS

I am running a storm cluster on AWS. But I want the storm cluster to expand automatically when the need comes. I figured out mesos is something like that. But I do not have much knowledge about mesos and its deployment on AWS.

Can mesos on AWS automatically increase the parallelism of my topology tasks by launching new instances and shutting them down when not necessary? If it can, how do we configure mesos for the same.

Upvotes: 2

Views: 274

Answers (2)

Adam
Adam

Reputation: 4322

Mesos does not directly handle autoscaling itself, but allows frameworks running on top of it to receive new resource offers and react to them by launching new task instances. I haven't used it personally, but you could try the Storm-Mesos framework for running Storm on Mesos: https://github.com/mesos/storm

Once you have Storm running on Mesos, ready to launch new instances as resources come available, you're ready to autoscale within the existing cluster's capacity. You'll probably want to take advantage of Amazon's Auto-Scaling Groups (ASGs) to scale up the number of Mesos nodes based on your need. As the ASG scales up more Mesos nodes, the resources from those nodes will be automatically offered to the Storm-Mesos framework, which can launch more Storm instances.

Upvotes: 1

Michael Hausenblas
Michael Hausenblas

Reputation: 13941

Yes, you're heading in the right direction. However I'd suggest to use Marathon rather than the low-level Mesos API.

See for example the GitHub repo obaidsalikeen/storm-marathon, which is particularly well done in terms of completeness and documentation richness.

Upvotes: 0

Related Questions