CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-ec2aws-lambda
basigow
basigow

Reputation: 165

AWS consuming data from API REST

I'm writing to you because I'm quite a novice with AWS... I only worked before with EC2 instances for simple tasks...

I am currently looking for an AWS service for reciving data using REST API calls (to external AWS services).

So far I have used EC2 where I deployed my library (python) that made calls and stored data in S3.

What more efficient ways does AWS offer for this? some SaaS?

I know that they are still more details to know in order to choose a good services but I would like to know from where I can start looking.

Many thanks in advance :)

Upvotes: 1

Views: 2368

Answers (2)

Aram Maliachi
Aram Maliachi

Reputation: 223

I make API requests using AWS Lambda. Specifically, I leave code that makes requests, writes the response to a file and pushes the response object (file) to AWS S3.

You'll need a relative/absolute path to push the files to wherever you want to ingest. By default lambda servers current working directory is: /var/task but you may want to write your files to /tmp/ instead.

You can automate the ingestion process by setting a CloudWatch rule to trigger your function. Sometimes I chain lambda functions when I need to loop requests with changing parameters instead of packing all requests within a single function,

i.e.

  1. I leave the base request (parameterized) in one function and expose the function through an API Gateway endpoint.
  2. I create a second function to call the base function once for each value I need by using the Event object (which is the JSON body of a regular request). This data will replace parameters within the base function.
  3. I automate the second function.

Tip:

Lambda sometimes will run your requests inside the same server. So if you're continuously running these for testing the server may have files from past calls that you don't want, so I usually have a clean-up step at the beginning of my functions that iterates through my filesystem to make sure there are no files before making the requests.

Using python 3.8 as a runtime I use the requests module to send the request, I write the file and use boto3 to push the response object to an aws S3 bucket.

Upvotes: 1

gusto2
gusto2

Reputation: 12085

To invoke an external service you need some "compute resources" to run your client. Under compute resources in aws we understand ec2, ecs (docker container) or lambda (serverless - my favorite)

You had your code already running on EC2 so you should already know you need VPC with a public subnet and ip address to make an outbound call regardless the compute resource you choose

Upvotes: 1

Related Questions