CODEWITHSUNDEEP
pythonamazon-web-servicesaws-lambdaamazon-dynamodbconcurrent.futures
user2392965
user2392965

Reputation: 445

AWS Lambda Boto3 DynamoDB Concurrency

I am trying to use concurrent.futures to retrieve items from AWS DynamoDB asynchronously to shorten the function time.

def retrieve_dynamo_item(dynamo_tablename, key):
    dynamodb = boto3.resource('dynamodb')
    dynamo_table = dynamodb.Table(dynamo_tablename)
    dynamo_result = dynamo_table.get_item(Key=key)
    return dynamo_result.get('Item')

def lambda_handler(event, context):

    dynamo_queries = ## code
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        futures = [executor.submit(retrieve_dynamo_item, dynamo_table, key) for dynamo_table, key in dynamo_queries]

    final_result = []
    for f in concurrent.futures.as_completed(futures):
        if f.result():
            final_result += f.result().get('result')

However when deploying and testing the lambda function performance does not seem to be improved but in fact worse than just running the queries synchronously. Any suggestions on what could be the reason?

Upvotes: 0

Views: 864

Answers (2)

Jens
Jens

Reputation: 21530

As Oleksii already said, the default memory setting of Lambda will only give you 1 CPU. So concurrency is kind of hard to achieve. The slower performance might be due to the added overhead of handling threads etc.

The following table contains the mapping of memory to CPU cores at this moment (could change in the future).

Memory CPU cores
128 - 1769 1 
1770 - 3538 2 
3539 - 5307 3 
5308 - 7076 4 
7077 - 8845 5 
8846 - 10240 6 

So you could increase your Lambdas memory to 2000MB for example and check if that improves performance.

Furthermore, I'd definitely recommend using Batch operations instead of just GetItem.

Upvotes: 3

alexis-donoghue
alexis-donoghue

Reputation: 3387

Parallel executions are most likely impacted by computing power available to the function. With default memory allocation you don't have much of CPU. See this answer for more details

Upvotes: 1

Related Questions