CODEWITHSUNDEEP
pythonaws-glue
user1411837
user1411837

Reputation: 1564

AWS Glue - Python Shell Jobs Secret Manager Connectivity Issues

I am using Python Shell Jobs under AWS Glue which has boto3 and a few other libraries built-in . I am facing issues trying to access the secrets manager to get credentials to my RDS instance running Mysql , the job keeps running forever without any (error/success) message nor does it time out .

Below is the simple code that runs even from my local or a lambda for Python3.7 but not in Python Shell GLUE , 

import boto3
import base64
from botocore.exceptions import ClientError

secret_name = "secret_name"
region_name = "eu-west-1"

session = boto3.session.Session()

client = session.client(
    service_name='secretsmanager',
    region_name=region_name
)

get_secret_value_response = client.get_secret_value(SecretId=secret_name)
print(get_secret_value_response)

Would be very helpful if someone could point out if anything needs to be done additionally in Python Shell jobs under AWS Glue in order to access the secret manager credentials .

Upvotes: 2

Views: 2569

Answers (2)

Karan Hebbar
Karan Hebbar

Reputation: 176

When you create a job without any VPC configuration , then glue tries to reach the secret manager through internet , if the policies allows to have internet route then we can connect to secret manager

But when a glue job is created with VPC configuration/connection then all the request are made from your VPC/subnet where the connection points to , if this is the case, make sure you have secret manager endpoint present in your route table of the subnet where glue launches the resources.

https://docs.aws.amazon.com/glue/latest/dg/setup-vpc-for-glue-access.html

https://docs.aws.amazon.com/secretsmanager/latest/userguide/vpc-endpoint-overview.html

Upvotes: 1

Abdelrahman Maharek
Abdelrahman Maharek

Reputation: 862

Make sure the IAM role used by the Glue Job has the policy SecretsManagerReadWrite

Also AWSGlueServiceRole and AmazonS3FullAccess According to the documentation

Upvotes: 1

Related Questions