CODEWITHSUNDEEP
pythonboto3workflowetlaws-glue
Gustavo Zilles
Gustavo Zilles

Reputation: 1

How to create a workflow in AWS Glue using Boto3 library?

I'm having problems to create a ETL workflow in AWS Glue using Boto3 library with all orchestration include. I created all jobs and triggers but when I use glue.create_workflow(Name="Test) command there's no way to include orchestration.

I tried to use this parameter

DefaultRunProperties={
        'string': 'string'
    }

With:

'--extra-files': 'file' #Like when you use a external libray in a job

'Graph': {'Nodes': [], 'Edges': []}} #Graph output using glue.get_workflow('Test')

Is this possible or you can only do this using AWS Console?

Upvotes: 0

Views: 757

Answers (1)

SimonClo
SimonClo

Reputation: 1

I had the same issue and did not find any solution directly from the Boto3 module.

However, I discovered AWS Glue Blueprint which allows you to create workflows from a template and therefore to duplicate them easily with a few input parameters. It is then possible to start this Blueprint from Boto3 by giving it the right inputs.

The official AWS documentation on this subject and their tutorial directory: https://docs.aws.amazon.com/glue/latest/dg/orchestrate-using-workflows.html https://github.com/awslabs/aws-glue-blueprint-libs

Hopefully this will help you

Upvotes: 0

Related Questions