CODEWITHSUNDEEP
amazon-web-servicesparallel-processingflashbotoamazon-swf
user1167910
user1167910

Reputation: 145

large number of parallel task causes aws swf stop

I used boto amazon swf api to do testing.

On this page http://boto.readthedocs.org/en/latest/swf_tut.html, it has a parallel workflow example called "Parallel Activity Execution", it has 5 activities parallel at one level.

However, I tested to do 500 parallel for this example. It doesn't work. Workers will stop after about 40-50 seconds and terminate.

Can someone tell me why I can't run large parallel jobs?

Upvotes: 0

Views: 369

Answers (2)

Hevy
Hevy

Reputation: 101

If, you look at the examples, you don't always return true. Workers and deciders only return true when they have to do something in these examples. So, your 500 workers because they did not have anything to do.

Upvotes: 0

Maxim Fateev
Maxim Fateev

Reputation: 6870

Each workflow in SWF has limited capacity. To scale out your application you have to partition your load across multiple workflow executions. For example if the top level workflow starts 100 child workflows and each child workflow starts 100 activities you end up with 10k activities running in parallel.

Upvotes: 1

Related Questions