CODEWITHSUNDEEP
socketslambdaaws-lambda
0xtuytuy
0xtuytuy

Reputation: 1654

Lambda + API Gateway: Long executing function return before task finished

I have a function in Lambda that executes for up to 30s depending on the inputs. That function is linked to an API gateway so I am able to call it through POST, unfortunatly the API Gateway is limited to 30s exaclty, so if my function runs for longer I get an "internal server error" back from my POST.

I see 2 solutions to this:

The function is coded in Python3.6 I therefore have access to the event and context object.

Upvotes: 2

Views: 3508

Answers (5)

acgawade
acgawade

Reputation: 1

To address the issue of API Gateway's 30-second timeout, you can use AWS Lambda Function URLs, which provide a direct HTTP(S) endpoint to invoke your Lambda function.

Unlike API Gateway, Lambda Function URLs allow you to configure a timeout of up to 15 minutes, making them ideal for functions that may run longer. You can setup an open or IAM-based authentication depending on your security needs. If immediate acknowledgment of the request is acceptable, you can configure the Lambda function to process the task asynchronously and quickly return a response while continuing execution in the background.

As you have a use case of requiring progress tracking, you can store the function's status updates in a service like DynamoDB and retrieve them separately. This solution simplifies the process by bypassing API Gateway, extends execution time flexibility, and reduces complexity, ensuring your Lambda function runs without time constraints.

Upvotes: 0

grantr
grantr

Reputation: 1050

Here's a real example of executing one lambda function asynchronously from another lambda function:

import json
import boto3


def lambda_handler(event, context):
    
    payload = {}
    
    ##take the body of the request and convert to string    
    body = json.loads(event['body'])
    
    ##build new payload with body and anything else your function downstream needs
    payload['body'] = json.dumps(body)
    payload['httpMethod'] = 'POST'
    payloadStr = json.dumps(payload)
    
    client = boto3.client('lambda')
    client.invoke(
        FunctionName='container name',
        InvocationType="Event",
        Payload=payloadStr
    )


    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

In this example, the function invokes the other one and returns right away.

invoke documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda.html#Lambda.Client.invoke

Upvotes: 0

balsick
balsick

Reputation: 1201

statut approach is nice, but I'd like to give you an alternative: AWS step functions. Basically it does what statut's solution does but it might be more easy to deploy.

All you have to do is to create an AWS Step Functions state machine that has your lambda function as its only task. AWS Step functions intrinsically works asynchronously, and has methods to know the status and, eventually, the result. Check the docs for more info.

Upvotes: 5

statut
statut

Reputation: 909

Return a notification to api gateway before the lambda function is actually finished. This is acceptable for my use case, but I am not sure again how would this work.

Unfortunately, you will not be able to return a result until the Lambda is finished. Otherwise, AWS will interrupt execution of Lambda if you try to do it, for example, via multithreading.

I suggest you create the following system:

  1. API Gateway puts a request to Kinesis stream and returns a response of successfully putting. The response will contain SequenceNumber of the Kinesis Record. (How to do it)
  2. Lambda is invoked by Kinesis Stream Event, processes the request and saves that your job is done (or completed result) to DynamoDB with SequenceNumber as id.
  3. You invoke another API Gateway method and SequenceNumber to check the status of your job in DynamoDB (if necessary).

If you need just to run the Lambda without knowing of job result you can avoid using DynamoDB.

Upvotes: 6

cementblocks
cementblocks

Reputation: 4596

A very simple solution would be for your lambda function to just asynchronously invoke another lambda function and then return a result to API Gateway.

Upvotes: 2

Related Questions