CODEWITHSUNDEEP
amazon-s3aws-lambdaamazon-redshiftaws-batch
user3400887
user3400887

Reputation: 409

Load S3 data into Redshift using batch operations job

I want to load data stored in S3 bucket into Redshift. I know it can be done manually. But, is it possible to use S3 batch operation job and schedule this task of loading S3 data into Redshift, on a regular interval?

Upvotes: 1

Views: 1579

Answers (3)

Shailesh
Shailesh

Reputation: 2286

You can write a Python script to do this by executing the COPY command to load data from S3 to Redshift. Use the psycopg2 library in Python to run SQL commands on Redshift via Python code.

Upvotes: 0

demircioglu
demircioglu

Reputation: 3465

You can invoke a Lambda function from S3 Batch operations to load data into Redshift.

Check Invoking AWS Lambda Functions in Jeff Barr's below post

https://aws.amazon.com/blogs/aws/new-amazon-s3-batch-operations/

Upvotes: 2

Julien Simon
Julien Simon

Reputation: 2729

AWS Data Pipeline is a managed service that does exactly that. You can configure and schedule batch jobs that import data from S3 to Redshift.

Here's a tutorial: https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-copydata-redshift.html

Upvotes: 1

Related Questions