CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaaws-step-functionsaws-batch
user12073121
user12073121

Reputation: 81

AWS Step Functions invoke Dynamic Parallel Step Functions

I'm solving a problem using step function workflows.

The problem goes like this, I have a worklow of 10 AWS Batch jobs.

The first 3 jobs run in sequence and the 4-7 jobs are dynamic steps i.e, they needs to run multiple times with different parameters as specified.

And for each 4-5-6-7 workflow, there are multiple executions of 8-9-10 jobs based on number of parameters.

Looks like Map is best possible fit here but if any of the job fails in map state of 4-5-6-7 entire step fails. I don't want one execution to effect the other execution.

Approach: I have designed 3 step functions. First step function runs 1-3 jobs and last step calls for a lambda function which submits multiple executions of 4-5-6-7 jobs. And for each 4-5-6-7 execution another lambda gets triggered to submit multiple executions of 8-9-10 jobs.

I'm connecting the step functions manually through lambda functions.

Is this the correct approach or are there better ways of doing it?

Upvotes: 0

Views: 1941

Answers (1)

Rob S.
Rob S.

Reputation: 469

I'd suggest a couple more elements to make your solution more production-ready.

First, I would suggest that you eliminate the Lambda function calls and use the "Run a Job" (.sync:2) service integration for Nested workflows. I just did a Twitch episode on this yesterday.

Second, if you want to continue after a failed execution inside your Map State, make sure that you are implementing Catchers (and optionally Retriers). I did a Twitch episode on this last Tuesday, and there's some discussion of error handling in the first video linked above.

So for your specific case, I suggest you:

  1. start by making the 8-9-10 steps into an independent workflow (Child A)
  2. invoke Child A from steps 4-5-6-7 via the "Run a Job" service integration inside a Map State
  3. migrate steps 4-5-6-7 into an independent workflow (Child B)
  4. invoke Child B from the parent workflow (steps 1-2-3), again via the "Run a Job" service integration

For more information on parallelism in Step Functions and Lambda functions, see this Twitch episode.

Code samples for all of the above are available in this repo on GitHub.

I am contributing this on behalf of my employer, Amazon. My contribution is licensed under the MIT license. See here for a more detailed explanation.

Upvotes: 2

Related Questions