CODEWITHSUNDEEP
pythonmultithreadingamazon-web-servicesamazon-s3
Sebastian Łącki
Sebastian Łącki

Reputation: 197

Multithreaded aws function

I asked a related question on creating parallel processes in a python lambda function and I'm more curious on how to tackle a specific issue on max_pool_sizes. I'm using the threading module and in short my issue is trying to access multiple files from S3 to retrieve data from all of them simultaneously. Currently it seems that there is a max_pool_size attribute that is preventing me from getting everything I need. As a result I get this error Connection pool is full, discarding connection: s3.amazonaws.com Has anyone else had a similar error or have any suggestions?

Upvotes: 1

Views: 1211

Answers (2)

Renato Byrro
Renato Byrro

Reputation: 3784

I had a similar need and found a solution on the AWS Compute blog. I'm linking below for future readers of this thread:

Parallel Processing in Python with AWS Lambda (by Oz Akan, Sep/2017, AWS Compute blog)

Upvotes: 0

MERT TOSLALI
MERT TOSLALI

Reputation: 11

You may examine ClientConfiguration class and use setMaxConnections(int maxConnections) method for this purpose. Take a look on this link:

Upvotes: 1

Related Questions