CODEWITHSUNDEEP
amazon-web-servicesapache-sparkaws-glue
karthikeayan
karthikeayan

Reputation: 5000

Does AWS Glue Jobs have any relation with Dev Endpoints?

Yesterday when I was trying to run my Glue job which connects to external third party service which is accessible in internet, Glue job was throwing "connect time out" error.

When I checked the internet access in my Dev Endpoint it doesn't had internet access. Then I added Nat Gateway and attached it to the route table of subnet which is configured in the dev endpoint to give internet access.

After that my Glue jobs are able to connect to the external third party service.

Does Glue Jobs have any relation with the Dev Endpoint? Or both are completely isolated?

Upvotes: 0

Views: 303

Answers (1)

Eman
Eman

Reputation: 851

S3 to S3 jobs are isolated will not connect to internet. A devendpoint with no networking information on setup will connect to the internet. If you setup a devenpoint with a connection, the connection will need a route to the internet, i.e an internet gateway. If you want your job to connect to a service on the internet or in another vpc then add jdbc connection to the job to make this happen. This will allow Glue to use the networking on your connection and launch elastic network interfaces that facilitate the communication with the service that you want.

a simple test in python: 

os.system("nc -vz google.com 443")

nc: connect to google.com port 443 (tcp) failed: Connection timed out
nc: connect to google.com port 443 (tcp) failed: Network is unreachable

In a devendpoint with no networking

>>> import os
>>> os.system("nc -vz google.com 443")
Connection to google.com 443 port [tcp/https] succeeded!

Upvotes: 1

Related Questions