Passing parameters to Glue Job using Step Function

Question

I have a Step function that enables my glue jobs to synchronously run by passing multiple parameters from event bridge which contains the job that will be running and its arguments but when I look to my glue they are running at the same time.

{
  "Comment": "A description of my state machine",
  "StartAt": "Pass",
  "States": {
    "Pass": {
      "Type": "Pass",
      "Next": "Map"
    },
    "Map": {
      "Type": "Map",
      "Iterator": {
        "StartAt": "Glue StartJobRun_1",
        "States": {
          "Glue StartJobRun_1": {
            "Type": "Task",
            "Resource": "arn:aws:states:::glue:startJobRun.sync",
            "Parameters": {
              "JobName.$": "$.job_name",
              "Arguments.$": "$.Arguments"
            },
            "End": true
          }
        }
      },
      "ItemsPath": "$.detail.config",
      "End": true
    }
  }
}

The first glue job should finish first before I proceed with another job. Can you suggest what I can do to run them synchronously in sequence

{
  "config": [
    {
      "job_name": "dev_1",
      "Arguments": {
        "--environment": "dev"
      }
    },
    {
      "job_name": "dev_2",
      "Arguments": {
        "--environment": "dev"
      }
    }
  ]
}

Answer 1

The Map state in your Step Functions workflow takes the input array and executes your states in the iterator in parallel (default 40 concurrent iterations).

To execute the Glue jobs in sequence, add "MaxConcurrency": 1 to the Map state. This will process items in the array synchronously and sequentially in the order of appearance.

Here's the modified Step Functions workflow definition

{
  "Comment": "A description of my state machine",
  "StartAt": "Pass",
  "States": {
    "Pass": {
      "Type": "Pass",
      "Next": "Map"
    },
    "Map": {
      "Type": "Map",
      "Iterator": {
        "StartAt": "Glue StartJobRun_1",
        "States": {
          "Glue StartJobRun_1": {
            "Type": "Task",
            "Resource": "arn:aws:states:::glue:startJobRun.sync",
            "Parameters": {
              "JobName.$": "$.job_name",
              "Arguments.$": "$.Arguments"
            },
            "End": true
          }
        }
      },
      "ItemsPath": "$.detail.config",
      "End": true,
      "MaxConcurrency": 1
    }
  }
}