Deploy StepFunctions with CloudFormation, from external definition file

Question

I'm trying to deploy stepfunctions with CloudFormation, and I'd like to reference the actual stepfunction definition from an external file in S3.

Here's how the template looks like:

StepFunction1: 
    Type: "AWS::StepFunctions::StateMachine"
    Properties:
      StateMachineName: !Ref StepFunction1SampleName
      RoleArn: !GetAtt StepFunctionExecutionRole.Arn
      DefinitionString:  
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: 
              Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunction1/definition.json'

However, this doesn't seem to be supported, as we are getting error

Property validation failure: [Value of property {/DefinitionString} does not match type {String}]

I am doing something similar for APIs, referencing the actual API definition from an external swagger file, and that seems to be working fine.

Example:

SearchAPI:
    Type: "AWS::Serverless::Api"
    Properties:
      Name: myAPI
      StageName: latest
      DefinitionBody: 
        Fn::Transform:
          Name: AWS::Include
          Parameters:            
            Location: 
              Fn::Sub: 's3://${ArtifactsBucketName}/ApiGateway/myAPI/swagger.yaml'

How can I make this work?

Answer 1

Here you can find another approach to this problem: https://github.com/aws-samples/lambda-refarch-imagerecognition

And here is a brief summary:

As part of your CloudFormation template file (let's say it's named template.yml) you define the state machine as follows:

  LambdaA:
    ...
  LambdaB:
    ...
  LambdaC:
    ...
  MyStateMachine:
    Type: AWS::StepFunctions::StateMachine
     Properties:
       StateMachineName: !Sub 'MyMachine-${SomeParamIfNeeded}'
       DefinitionString:
         !Sub
         - |-
         ##{{STATEMACHINE_DEF}}
         - {
          lambdaArnA: !GetAtt [ LambdaA, Arn ],
          lambdaArnB: !GetAtt [ LambdaB, Arn ],
          lambdaArnC: !GetAtt [ LambdaC, Arn ]
         }
         RoleArn: !GetAtt [ StatesExecutionRole, Arn ]

Then you will need replacement script, here you can find one: inject_state_machine_cfn.py

Then you can create your state machine definition as a separate file, say state_machine.json with a content:

{
  "Comment": "This is an example of Parallel State Machine type",
  "StartAt": "Parallel",
  "States": {
    "Parallel": {
      "Type": "Parallel",
      "Next": "Final State",
      "Branches": [
        {
          "StartAt": "Run Lambda A",
          "States": {
            "Run Lambda A": {
              "Type": "Task",
              "Resource": "${lambdaArnA}",
              "ResultPath": "$.results",
              "End": true
            }
          }
        },
        {
          "StartAt": "Run Lambda B",
          "States": {
            "Run Login Functionality Test": {
              "Type": "Task",
              "Resource": "${lambdaArnB}",
              "ResultPath": "$.results",
              "End": true
            }
          }
        }
      ]
    },
    "Final State": {
      "Type": "Task",
      "Resource": "${lambdaArnC}",
      "End": true
    }
  }
}

Once you have all these elements you can invoke the transformation function:

python inject_state_machine_cfn.py -s state_machine.json -c template.yml -o template.complete.yml

Answer 2

In the mean time, one solution could be to use yaml instead of json, to store the stepfunction definition externally. No string escaping:

DefinitionString: 
  Fn::Sub: |
    {
      "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
      "StartAt": "HelloWorld",
      "States": {
        "HelloWorld": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}",
          "End": true
        }
      }
    }

Answer 3

The trick is to escape the StepFunction DefinitionString property, and include the actual property, DefinitionString, in the external CloudFormation referenced file. Escaping only the stepfunction definition string would fail, CloudFormation complaining that the referenced Transform/Include template, is not a valid yaml/json. Here's how it looks like: Template:

StepFunction1: 
    Type: "AWS::StepFunctions::StateMachine"
    Properties:
      StateMachineName: !Ref StepFunction1SampleName
      RoleArn: !GetAtt StepFunctionExecutionRole.Arn      
      Fn::Transform:
        Name: AWS::Include
        Parameters:
          Location: 
            Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunction1/definition.json'

External stepfunction definition file:

{
    "DefinitionString" : {"Fn::Sub" : "{\r\n  \"Comment\": \"A Retry example of the Amazon States Language using an AWS Lambda Function\",\r\n  \"StartAt\": \"HelloWorld\",\r\n  \"States\": {\r\n    \"HelloWorld\": {\r\n      \"Type\": \"Task\",\r\n      \"Resource\": \"arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}\",      \r\n      \"End\": true\r\n    }\r\n  }\r\n}"}
}

Now, although this solves the problem, it's a bit more difficult to maintain the StepFunction definition, in this form, in source control.

So I've thought about using a CloudFormation custom resource backed by a lambda function. The lambda function would deal with the actual StepFunction DefinitionString escaping part.

Here's how it looks like: Template:

StepFunctionParser:
    Type: Custom::AMIInfo
    Properties:
      ServiceToken: myLambdaArn
      DefinitionString: 
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: 
              Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunctionX/definition.json'   
  StepFunctionX: 
    Type: "AWS::StepFunctions::StateMachine"
    Properties:
      StateMachineName: StepFunction1SampleNameX
      RoleArn: !GetAtt StepFunctionExecutionRole.Arn      
      DefinitionString: !GetAtt StepFunctionParser.DefinitionString

External StepFunction definition file:

{
  "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
  "StartAt": "HelloWorld",
  "States": {
    "HelloWorld": {
      "Type": "Task",
      "Resource": {"Fn::Sub" : "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}" },
      "End": true
    }
  }
}

Here's the documentation for creating AWS Lambda-backed Custom Resources.

There's still a problem with this. Transform/Include converts external template boolean properties into string properties. Therefore, DefinitionString

"DefinitionString": {
            "States": {
                "HelloWorld": {
                    "Type": "Task",
                    "Resource": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}",
                    **"End": true**
                }
            },
            "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
            "StartAt": "HelloWorld"
        }

becomes

"DefinitionString": {
            "States": {
                "HelloWorld": {
                    "Type": "Task",
                    "Resource": _realLambdaFunctionArn_,
                    **"End": "true"**
                }
            },
            "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
            "StartAt": "HelloWorld"
        }

CloudFormation then complains about the StepFunction definition not being valid:

Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: Expected value of type Boolean at /States/HelloWorld/End' 

Is this a CloudFormation Transform/Include issue? Can someone from AWS give a statement on this?