CODEWITHSUNDEEP
python-2.7amazon-web-servicesamazon-s3
narayanan s
narayanan s

Reputation: 111

Python script to load data from AWS S3 to Redshift

Has anybody worked on creating a python script to load data from s3 to redshift tables for multiple files. How can we acheive it in AWS CLI. Your learnings and inputs on the same is appreciated.

Upvotes: 1

Views: 2497

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 269081

The COPY command is the best way to load data from Amazon S3 to Amazon Redshift. It can load multiple files in parallel into the one table.

Use any Python library (eg PostgreSQL + Python | Psycopg) to connect to Amazon Redshift, then issue the COPY command.

The AWS Command-Line Interface (CLI) does not have the ability to run the COPY command on Redshift because it needs to be issued to the database, while the AWS CLI issues commands to AWS. (The AWS CLI can be used to launch/terminate a Redshift cluster, but not to connect to the cluster itself.)

Upvotes: 2

Related Questions