CODEWITHSUNDEEP
amazon-web-servicesapiaws-api-gatewayaws-glue
Indi
Indi

Reputation: 429

Is there a way to execute a glue job via API Gateway?

I want to know that is there any way to execute an AWS Glue Job (Python shell type) directly via API Gateway. My Glue job reads a file from the AWS S3 bucket and does an operation using the file content. I have already created the python script in the AWS glue job and it is run successfully. I want to execute the Glue Job via API Gateway.

Is it possible to do so?

Can you please give me a hint to do this? Thanks in advance!

Upvotes: 4

Views: 4515

Answers (2)

JD D
JD D

Reputation: 8097

Yes, it is possible to invoke any AWS API in API Gateway via the AWS Proxy mechanism.

Building from what Marcin pointed you at, click here for a guide about the general ability to invoke AWS APIs via API Gateway

Specifically, you are going to want to target the StartJobRun action of the Glue Jobs API

Basically, you need to read the documentation to understand how AWS's StartJobRun REST API is invoked and pass that payload to your API Gateway and the API Gateway will happily forward the request to their own Glue Jobs REST API.

The upside to this is you have a direct integration without needing other infrastructure and the latency that comes with jumps through those additional hoops.

The downside is you have no control over the input/output to/from your API and you have to pass exactly what AWS wants you to pass to invoke their own REST APIs. Also, I personally find AWS's REST API documentation much more difficult to understand and more difficult to use than their Python and Node SDKs. That is why many folks create a simple lambda component in between the two services so they can define the request and response formats (which you can make very simple for clients) and you can use an SDK you are already familiar.

Upvotes: 3

Marcin
Marcin

Reputation: 238081

You can setup Lambda Proxy Integrations for the API gateway. The integration is the easiest and the recommended way of invoking lambda using API gateway.

So the only thing you need to do is basically to write your lambda that starts the glue job. You can also return some useful information to the end user through the api gateway. Maybe Glue job id or something like this.

Good tutorial explaining how to create the Lambda integration is here;

Upvotes: 3

Related Questions