CODEWITHSUNDEEP
amazon-web-servicesaws-step-functions
Iván E. Sánchez
Iván E. Sánchez

Reputation: 1263

Step Functions - Access State from previous Map Iteration

How can I get the results from previous Map Iterations in the next iteration when using MaxConcurrency: 1 in Amazon Step Functions?

Here's an example of the code I have

{
  "StartAt": "UploadUsers",
  "States": {
    "UploadUsers": {
      "Type": "Map",
      "MaxConcurrency": 1,
      "ItemsPath": "$.data.users",
      "Parameters": {
        "data.$": "$$.Map.Item.Value.data",
        "friends.$": "$.?????? Get created users ids"
      },
      "Iterator": {
        "StartAt": "UploadUser",
        "States": {
          "UploadUser": {
            "End": true,
            "Parameters": {
              "FunctionName": "${FnUploadUser}",
              "Payload": {
                "data.$": "$.user_data",
                "friends.$": "$.??????"
              }
            },
            "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
            "ResultPath": "$.data. ???",
            "Type": "Task"
          }
        }
      },
      "End": true,
      "ResultPath": "$.data.UploadUsers",
      "ResultSelector": {
        "result.$": "$"
      }
    }
  }
}

Suppose FnUploadUser is a lambda that returns the id of the created user. And I want to get the ids of the previously created users and use that value for the next user I'm about to create.

Upvotes: 1

Views: 1839

Answers (1)

fedonev
fedonev

Reputation: 25639

You can't. Map State iterations don't share state. Two workarounds:

(1) Manage the shared state externally: Each Map iteration writes and reads from, say, a DynamoDB table.

(2) Refactor to a "for" loop and keep the shared state in the execution output.

  1. Instead of using Map, insert a Choice State (after UploadUser) that checks for a "done" condition. If "done", finish, else loop back to UploadUser.
  2. UploadUser accepts the user_data array as input. It appends its output to, say, the uploaded output array.
  3. Each UploadUser iteration identifies the next user_data item by comparing it to the uploaded array. The iteration that processes the last item can also output done: true to signal to Choice that work is done.
  4. The Choice State loops back to UploadUser while there are more to process (i.e. while done is not present).

There are other ways to build steps 2-3. For instance, you could add next_item and total_items keys on the output to keep track of progress. The important point is that Choice loops until an exit condition is met.

Upvotes: 2

Related Questions