CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-lambdaamazon-redshiftamazon-cloudwatch
Aakash Basu
Aakash Basu

Reputation: 1767

Idea and guidelines on end to end AWS solution

I want to build an end to end automated system which consists of the following steps:

  • Getting data from source to landing bucket AWS S3 using AWS Lambda
  • Running some transformation job using AWS Lambda and storing in processed bucket of AWS S3
  • Running Redshift copy command using AWS Lambda to push the transformed/processed data from AWS S3 to AWS Redshift

From the above points, I've completed pulling data, transforming data and running manual copy command from a Redshift using a SQL query tool.

Doubts:

  • I've heard AWS CloudWatch can be used to schedule/automate things but never worked on it. So, if I want to achieve the steps above in a streamlined fashion, how to go about it?
  • Should I use Lambda to trigger copy and insert statements? Or are there better AWS services to do the same?
  • Any other suggestion on other AWS Services and of the likes are most welcome.

Constraint: Want as many tasks as possible to be serverless (except for semantic layer, Redshift).

Upvotes: 0

Views: 76

Answers (1)

danimal
danimal

Reputation: 1697

CloudWatch:

Your options here are either to use CloudWatch Alarms or Events.

With alarms, you can respond to any metric of your system (eg CPU utilization, Disk IOPS, count of Lambda invocations etc) when it crosses some threshold, and when this alarm is triggered, invoke a lambda function (or send SNS notification etc) to perform a task.

With events you can use either a cron expression or some AWS service event (eg EC2 instance state change, SNS notification etc) to then trigger another service (eg Lambda), so you could for example run some kind of clean-up operation via lambda on a regular schedule, or create a snapshot of an EBS volume when its instance is shut down.

Lambda itself is a very powerful tool, and should allow you to program a decent copy/insert function in a language you are familiar with. AWS has several GitHub repos with lots of examples too, see for example the serverless examples and many samples. There may be other services which could work for you in your specific case, but part of Lambda's power is its flexibility.

Upvotes: 1

Related Questions