CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-redshift
user14123081
user14123081

Reputation:

Is there a way to copy all objects inside a S3 bucket to Redshift using a Wildcard?

I have an S3 Bucket called Facebook

The structure is like this :

Facebook/AUS/transformedfiles/YYYYMMDDHH/payments.csv
Facebook/IND/transformedfiles/YYYYMMDDHH/payments.csv
Facebook/SEA/transformedfiles/YYYYMMDDHH/payments.csv

Is there a way to copy all payments.csv to AWS Redshift? something like :

copy payments Facebook/*/transformedfiles/YYYYMMDDHH/payments.csv

Upvotes: 0

Views: 302

Answers (1)

Parsifal
Parsifal

Reputation: 4486

No, because the FROM clause accepts an object prefix, and implies a trailing wildcard.

If you want to load specific files, you'll need to use a manifest file. You would build this manifest by calling ListObjects and programmatically selecting the files you want.

A manifest file is also necessary if you're creating the files and immediately uploading them, because S3 is eventually consistent -- if you rely on it selecting files with a prefix, it might miss some.

Upvotes: 1

Related Questions