CODEWITHSUNDEEP
aws-step-functions
J.S
J.S

Reputation: 159

How can you use Step Functions get execution hsitory to circumvent 25000 limit

I am trying to implement solution described here ,
https://www.readysetcloud.io/blog/allen.helton/when-not-to-use-step-functions/ to circumvent 25k limit. How do I get ARN for currently running stepfunctions machine dynamically and pass it to GetExecutionHistory step?

enter image description here

Upvotes: 0

Views: 405

Answers (2)

J.S
J.S

Reputation: 159

I was able to use "GetExecutionHistory" & fetch current executions ARN "AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.$": "$$.Execution.Id"

Upvotes: 0

Vilius Kukanauskas
Vilius Kukanauskas

Reputation: 139

This was a huge problem for us, so we implemented 2 different solutions, which I am glad to share.

First of all, we could not find the current step number in the AWS api.

A tutorial in AWS tells you, to implement the counter yourself. https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-continue-new.html

Which is what we did in one approach. Take care if you are counting inside a "Map" Step.

Every step in the Map gets the same input, so you have to use counter += MapIndex * countAmount. And use the output from very last step in the map. (just minor detail, that costed us few hours to find out)

I was not totally happy with this solution, as AWS definitely counts the steps somewhere. As they improved Step Functions over last months, I hope they will also increase the API and simply give us the number !

Second solution was : every step that was capable of being its own process, started its own stepfunction.

It is quite easy to start a child stepfunction, You can even start a process on some ec2 machine, and the main process will only continue, after you send specific key to AWS.

As every stepfunction gets its 25.000 limitation, we manage to split our data import process into multiple sub processes, which solved our problems.

Best of luck and I hope we will get the API call, to get the current process step in the near future.

Upvotes: 1

Related Questions