CODEWITHSUNDEEP
pythonamazon-web-servicesaws-lambdaserverless-frameworkserverless
Pedro Borges
Pedro Borges

Reputation: 1270

AWS Python lambda times out forever after first timeout

Problem:

I have a python lambda that constantly receives data every second and puts it into DynamoDB. I noticed that after the first time DynamoDB takes a little more and the function times out, all the following calls also timeout and it never recovers. The way to bring the lambda back to normal is to redeploy it. When it starts timing out, it does not display any logs. It times out without executing any of the code.

Below is a picture of our console that represents the issue.

aws lambda console

In order to reproduce the issue faster with this function I did the following:

  1. Redeploy it and see it is working fine.
  2. Reduce the memory available to the lambda to the minimum and timeout to 1 second. This will cause the first timeout
  3. Increase back the memory of the lambda to normal and even increase the timeout. However, the timeouts persist

Is there a way to resolve this issue without having to redeploy?

I have seen the same description of issue but with nodejs in this post: https://forums.aws.amazon.com/thread.jspa?threadID=234417.

I haven't seen any description related with the python lambda env

More information about the setup:

Lambda environments tested: python3.6 and python3.7

Tool to deploy lambda: serverless 1.57.0

serverless plugins used: serverless-python-requirements, serverless-wsgi

I am not using any VPC for the lambda

Thank you for the help,

Upvotes: 2

Views: 578

Answers (1)

Pedro Borges
Pedro Borges

Reputation: 1270

Figured out the trigger for the bug. When the lambda function zip uploaded is too large, after the first time it times out, it never recovers!

My solution was to carefully strip out the unnecessary dependencies to make the package smaller.

I created a repository using a docker container for people to reproduce the issue more easily:

https://github.com/pedrohbtp/bug-aws-lambda-infinite-timeout

Thanks for the messages in the comments. I appreciate whoever takes time to try to help here in SO.

Upvotes: 1

Related Questions