How do I pass JSON to an ECS Task in an AWS StepFunction?

Question

I am attempting to create an AWS StepFunctions workflow where I have a Lambda task followed by an ECS/Fargate task.

The Lambda takes an ID as an input and outputs some data in JSON form that is used by the ECS task, which is runs a Python script in its container environment. What I would like to do in StepFunctions is the following flow:

{ id: 1234 } -> [Lambda] -> { id: 1234, data: {...} }

{ id: 1234, data: {...} } -> [ECS] -> { id: 1234, result: "bar"}

For reference, here is an example configuration of an ECS Task: https://docs.aws.amazon.com/step-functions/latest/dg/sample-project-container-task-notification.html

I cannot figure out any way to pass the structured JSON input of a ECS Task to the container running the task.

Here are the things I have found so far:

• I can pass individual fields of a JSON input to the container by using JSONPath to select individual fields of the input and set them to environment variables. But if I assign the entire input object ($) to an environment variable, then it fails at runtime with a serialization error ([Object] cannot be converted to a string).
• I can create an intermediate lambda that takes the input and converts it to a JSON string that is stored in a single key-value in the output, then assign this single string key-value to an environment variable of ECS Task and parse it. However, this requires adding an entire extra Task and a few seconds of runtime + cost.

Here are some things I can't do:

• There doesn't seem to be any mechanism in boto3 to get the input of an existing ECS Task. I can get the input of an unassigned Activity, or I can get the input of the entire Execution. But there is no API for just getting the input of an existing, running Task, even though I have a Task Token.
• I cannot modify my original Lambda to output JSON as a string. I am using this result in multiple places (parallel tasks), and the other tasks are Lambdas that consume specific subfields of the output as their input.

What is the intended mechanism to pass a structured JSON object defined as the input to a Task to the executing container of an ECS/Fargate Task?

Answer 1

You can use pickle to serialize your JSON object into byte stream and encode it to string with base64. Pass the string to ECS as environment variable.

JSON---pickle-->Byte_Stream->--base64.encode-->String-->environment variable-->base64.decode-->Byte_Stream---pickle-->JSON

import base64
import pickle
if environ.get('SOME_JSON_BASE64') is not None:
    
 SOME_JSON=pickle.loads(base64.b64decode(environ.get('SOME_JSON_BASE64')))

To create the string for ENV,

import base64
import pickle

SOME_JSON=xxxx
print(base64.b64encode(pickle.dumps(SOME_JSON)))

Answer 2

You can use intrinsic functions to format the request before running the task:

const formatRequest = new sfn.Pass(this, 'FormatRequest', {
    parameters: {
        'request.$': 'States.JsonToString($)'
    }
})

Answer 3

Given you don't specify in the step that runs the Lambda a result path, the input of your container will be the output of your Lambda, that translates to:

"Overrides": {
   "ContainerOverrides": [
      {
        "Name": "container-name",
        "Environment": [
          {
            "Name": "SOME_ENV_VAR",
            "Value.$": "$"
          },

But even this is limited to what you can store as ENV, so you would need to make sure your JSON is actually a string

Answer 4

What is the intended mechanism to pass a structured JSON object defined as the input to a Task to the executing container of an ECS/Fargate Task?

Take a look at the Input and OutPut processing docs: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-input-output-filtering.html

This will help with you deciding what of the JSON input you want passed to the "Run Fargate Task" state (from the example you linked in you question)

Step Functions support the "RunTask" of ECS and a couple parameters: https://docs.aws.amazon.com/step-functions/latest/dg/connect-ecs.html

For Example,

1. Suppose I have my Lambda Function output this JSON

   {
       "commands": [
           "foo": { "bar" },
           "some command 1",
           "some command 2"
       ]
   }
   

2. I want my Run Fargate Task to haven an Input Path that only gets all of the input. In my state machine, after "Type": "Task", I will put:

   "InputPath":"$.commands",
   

3. Then in my "Parameters" for my Fargate Task after "NetworkConfigurations:{....}," I will place the Container Overrides that I want using the JSON Path syntax: https://github.com/json-path/JsonPath. However, I don't want all the input from the JSON, just the value of "foo"

   "Overrides": {
       "ContainerOverrides": [
           {
               "Name": "container-name",
               "Command.$": "$.commands.foo"
           }
       ]
   }
   

You can use the syntax used here: https://docs.aws.amazon.com/step-functions/latest/dg/connect-ecs.html