CODEWITHSUNDEEP
amazon-web-servicesasynchronousaws-lambdaboto3python-asyncio
Davi Barreira
Davi Barreira

Reputation: 1681

Invoke Lambdas asynchronously using boto3 locally

I've set a Lambda in AWS that reads an object in an s3 an does some processing. The thing is, I have many of such objects in the bucket, and I want to process all of them. Hence, I was planning on invoking the Lambda function locally using boto3:

functioname = 'lambda_handler'

for p in tqdm(payloads):
    client.invoke(
        FunctionName = functioname,
        InvocationType = 'Event',
        LogType = 'None',
        ClientContext = 'string',
        Payload = p
    )

The problem is, the code above is NOT async, the loop only moves on to the next case after a second or so. What is going on? Is there a work around this?

Upvotes: 0

Views: 336

Answers (1)

Coin Graham
Coin Graham

Reputation: 1584

In answer to your question "I have many of such objects in the bucket, and I want to process all of them.", the easiest way to achieve this is to enable the S3 Event trigger on the lambda that you want to process the files referencing the bucket where the files are located.

Then go into the bucket and select the files and copy them back into the bucket. That will trigger the S3 event for each file in its own lambda.

An added advantage is that any future files put into the bucket will also be processed going forward.

If you need to use the CLI to copy the files you can use a command like this:

aws s3 cp s3://<your bucket>/ s3://<your bucket> --recursive

Upvotes: 1

Related Questions