CODEWITHSUNDEEP
aws-glueaws-glue-workflow
Randomize
Randomize

Reputation: 9103

Is there any session available across AWS Glue Jobs?

In my AWS Glue Workflow (defined using CloudFormation) I have a few jobs (pythonshell) that have a repeated logic to create some parameters like the same db connection string etc. Also there is a repeated logic via Boto3 like this:

import boto3
sm_client = boto3.client(service_name='secretsmanager', region_name=region_name)
db_password = sm_client.get_secret_value(SecretId="MY_DB_PASSWORD")["SecretString"]

to fetch the relative password of the DB.

As said this sort of logic is repeated across multiple jobs. Is there any sort session or you have any idea about storing this sort of info across multiple jobs for the same workflow?

Upvotes: 0

Views: 155

Answers (1)

Vivek
Vivek

Reputation: 707

You can share information across Glue Jobs using workflow properties as defined at this https://docs.aws.amazon.com/glue/latest/dg/workflow-run-properties-code.html

But for sensitive information I would rather use SecretManager or Glue database connection.

Upvotes: 0

Related Questions