CODEWITHSUNDEEP
c#aws-lambda
Aniruddha
Aniruddha

Reputation: 1039

One or more errors occurred. (Exception of type 'System.OutOfMemoryException' was thrown.)

I'm running into below mentioned error when a list of more than 30k items is created, via Lambda function.

Error:

System.Collections.Generic.List`1.set_Capacity(Int32 value) at System.Collections.Generic.List`1.EnsureCapacity(Int32 min) at System.Collections.Generic.List`1.AddWithResize(T item) at

at System.Collections.Generic.List`1.set_Capacity(Int32 value)
   at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
   at System.Collections.Generic.List`1.AddWithResize(T item)
   at AWSLambda3.Function.ListS3ObjectsAsync(String bucketName, String filestoDwnld, IAmazonS3 client)

Code:

public async Task<List<string>> ListS3ObjectsAsync(string bucketName, string filestoDwnld, IAmazonS3 client)
        {
            //int fileCount = 0;
            List<string> pdfFiles = new List<string>();
            
            ListObjectsRequest request = new ListObjectsRequest
            {
                BucketName = bucketName,
                Prefix = filestoDwnld,
            };

            ListObjectsResponse response = await client.ListObjectsAsync(request);

            do
            {
                foreach (S3Object entry in response.S3Objects)
                    pdfFiles.Add(entry.Key);

                if (response.IsTruncated)
                {
                    request.Marker = response.NextMarker;
                }
                else
                    request = null;
                
            } while (request != null);

            return pdfFiles;
        }

I've also tried increasing list capacity, but that doesn't help as well. Please assist.

Upvotes: 1

Views: 1070

Answers (1)

Steve
Steve

Reputation: 216293

The reason for the OutOfMemoryException is in the infinite loop that is triggered when response.IsTruncated == true In this case the request is never set to null and the loop doesn't stop. The code restart a new loop loading again the same set of elements into the list pdfFiles and so on until you have no more memory.

I don't know exactly how your service backend works but I could immagine that you need to change just one line of code inserting the request to the service call inside the loop

do
{
    // This inside the loop will be executed at least one time and 
    // eventually again until the IsTruncated property is set to true.
    ListObjectsResponse response = await client.ListObjectsAsync(request);

    foreach (S3Object entry in response.S3Objects)
        pdfFiles.Add(entry.Key);

    if (response.IsTruncated)
    {
        request.Marker = response.NextMarker;
    }
    else
        request = null;
    
} while (request != null);

In this way, after the first loop you ask again a new set of elements from the service backend pointing at the NextMarker and eventually you will reach a point in which IsTruncated will be false ending the loop

Upvotes: 3

Related Questions