CODEWITHSUNDEEP
sqlpostgresqlamazon-redshift
John Chrysostom
John Chrysostom

Reputation: 3963

Incorporate File Name in Redshift COPY

I need to load ~2 million CSV files from an S3 bucket to a Redshift table. This would be easy (just use a COPY with a wildcard or a manifest file), except that I need to incorporate the name of each file into the resulting table. Suppose file1.csv and file2.csv both contain:

a,b,c
d,e,f

I want my table to have

file1 a b c
file1 d e f
file2 a b c
file2 d e f

Is there a way this can be accomplished with a single COPY statement? Or will I need to iterate through the list of files and load/insert them one at a time?

I suspect the latter option would be a massive performance hit...

Upvotes: 3

Views: 4931

Answers (1)

Trevor Morris
Trevor Morris

Reputation: 121

This is not currently possible.

Here is a brief thread on the AWS forum, with a response from AWS that the have "created a feature request" but "cannot provide an ETA on this": https://forums.aws.amazon.com/thread.jspa?messageID=590722&#590722

A similar question already on Stack Exchange: Redshift add column when importing with COPY

Upvotes: 4

Related Questions