CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-athenaaws-step-functions
Falcon
Falcon

Reputation: 442

Running Big Queries on Athena using Lambda

As we know that Lambda timeouts after 15 minutes of execution. I am looking for a way to execute an Athena query which is on a really huge dataset. Invoking the query from Lambda will introduce the problem of Lambda timing out before the query is processed. My ultimate goal is to invoke the query and to know when the query is complete so I could process the results. Could a state machine be used to query Athena? or does Lambda function by only invoking the athena and not waiting for results, I'll query the athena for results after the duration in which I expect the query to run? Any other ideas would be appreciated.

Upvotes: 2

Views: 3254

Answers (1)

jarmod
jarmod

Reputation: 78563

With Athena, you initiate a query and you get a unique Athena query execution ID back as a result. Athena executes the query asynchronously. Athena ultimately stores the results of the query in S3. You can later retrieve those results by making a 2nd API call passing in the original query execution ID.

See Working with Query Results, Output Files, and Query History.

Upvotes: 5

Related Questions