CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaconcurrencytimeout
user3553653
user3553653

Reputation: 127

Lambda timeouts with calls to external api when scaling up

I have set up a lambda A (Nodejs) which gathers data from other lambdas B,C,D,... by invoking them synchronously. These other lambdas call an external API. The issue is that when there is traffic, these other lambdas (B,C,D, ...) timeout their request to this external API. With a single invokation of Lambda A, these other lambdas perform their external api http request in under a second, but under load, the time keeps increasing.

The timeout is set at 10 seconds so reattempting can be made, but I tried changing this to a minute just to see if I get a response, and it does happen. It seems as if there is not enough of something available and subsequent requests get queued.

There are no errors in logs (regarding hitting AWS limits) and I'm not hitting AWS Concurrency limits. I also thought I could be hitting burst concurrency limits, but there are no errors in logs.

enter image description here

(Graph shows sum per minute)

How can the ConcurrentExecutions be so high if there are only 366 invocations?

I have started fiddling around with maxSockets (I wasn't aware that Infinity equals to only 50), but I'm thinking it is not set correctly at the moment. Could this however be the case, eq. it only can process 50 connections at a time and rest are queued?

Code below (mock).

const soapConfig = {
  endpoint: 'https://soapapi.com',
  options: { timeout: 29000, time: true, pool: { maxSockets: Infinity },
    wsdl: `${__dirname}/ASC2021.wsdl`
  }
}

const handler = (event, context, callback) => {
  let processed = (async function () {
    let params = Object.assign({}, event.pathParameters, event.queryStringParameters)
    const client = await soap.createClientAsync(soapConfig.wsdl,
      soapConfig.options, soapConfig.endpoint)
    const result = await client.GetData(params, soapConfig.options)
    return result
  })()
  postProcess(processed, context, callback)
}

AWS_NODEJS_CONNECTION_REUSE_ENABLED is set to 1

Could there some queue somewhere that I'm hitting?

EDIT: I forgot to add that I have had detailed discussions with the API maintainer and the calls do not even reach said api. If timeout is raised to said 1 minute, the api receives the request but near the end of the request. There are also no limits on the API.

Upvotes: 0

Views: 1484

Answers (1)

Lunarantic
Lunarantic

Reputation: 51

As mentioned under "Using Performance Metrics" at https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics.html you should be using "Maximum" metric not "Sum" for "ConcurrentExecutions". Using "Sum" will not make sense.

By what you explained your use-case there seems to API limit that you can call per second to external API (maybe 50 requests per second as per your information). When you are exceeding the limit your lambda retries resulting into longer execution times, eventually timing out under load. You can leverage ReservedConcurrency feature of lambda to limit your lambda to exceed external api limit. And, apply some backoff logic on throttling errors codes to be graceful to your downstream services.

Upvotes: 1

Related Questions