CODEWITHSUNDEEP
aws-glueaws-step-functions
NikRED
NikRED

Reputation: 1195

How to run a state (glue job) based on condition in step function?

I have a step function contains 2 glue jobs one after other. Is there a way to run second glue job based on a value from the first glue job?

can I use input/output options in the step function for this requirement? if so, how?

Thanks, Nikhil

Upvotes: 1

Views: 588

Answers (1)

ms12
ms12

Reputation: 581

If I understood you correctly, you want to pass data between Glue jobs using one's output as the second one's input. You can't.

First, Glue jobs don't output the data like a Lambda function. Second, even if they did, Step Function has a hard limit of 6MB you can send between the steps, which I am sure wouldn't be enough for you.

The way you can do it is have the first Glue job write its output to a specific location in S3, and use that location as a source for your second Glue job that will run when the first finishes.

Upvotes: 1

Related Questions