CODEWITHSUNDEEP
pythonpostgresqlamazon-web-servicesaws-lambdaamazon-redshift
Sharad Sharma
Sharad Sharma

Reputation: 51

How to asynchronously pass redshift query from lambda function?

I want to execute a group of copy commands in redshift from a lambda function where I copy around 100 GB of files from S3 to a table in redshift. I cannot use Redshift Data API for this (because I cannot have a secret arn for the cluster now and getting temporary credentials is also not ideal in my case).

I have tried using the psycopg2 library but as soon as the lambda function timeouts, the execution stops too.

Is there any way I can asynchronously pass the queries to redshift and when the lambda function timeouts, the query still keeps executing in redshift?

Upvotes: 0

Views: 761

Answers (1)

Bill Weiner
Bill Weiner

Reputation: 11032

Sorry, no. This is fundamental to ODBC / JDBC connections - if the connection is dropped the transaction will be aborted. I went so far a few years back as to have a small EC2 repeat Lambda SQL to Redshift so that Lambda could end w/o closing the connection. Worked great but having a server in the center of a server-less solution wasn’t quite right. You could go down this right if you like.

This is why Redshift Data API is a big step forward

Upvotes: 2

Related Questions