CODEWITHSUNDEEP
amazon-web-servicesrestaws-glue
Maanoj Kumar
Maanoj Kumar

Reputation: 53

AWS Glue and REST API

I am currently looking at ETL services to use to transform and push data to a data warehouse. I have come across AWS Glue and I am wondering if it is possible to extract data from external sources using REST API with AWS Glue. I have gone through the documentation and have not found out if it is possible. Does anyone know if this is possible and have any resources which I can refer to to point me in the right direction ?

Upvotes: 2

Views: 3777

Answers (2)

H Kaushal
H Kaushal

Reputation: 11

We had a similar scenario where we need to load some data from REST API. First, We wrote AWS Glue Python Shell based JOB to load the data from REST API to S3 Bucket. And then, used AWS Glue to read from S3 using ATHENA and load to our relational database.

Upvotes: 1

Marcin
Marcin

Reputation: 238081

As you already wrote, such functionality is not supported by AWS glue. Instead you could dump your data to S3, and then use glue to ETL it your warehouse. This would require you to write custom solution that would access your source data through the REST API and store it in S3.

Alternatively, instead of storing the data in S3 you could stream it and process using Streaming ETL Job. In either case, you would have to develop some code to interact with your REST API and get the data you need. Thus, you could maybe just skip Glue entirely and load your data directly to your warehouse.

Upvotes: 2

Related Questions