CODEWITHSUNDEEP
amazon-dynamodb
tony09uk
tony09uk

Reputation: 2991

IDynamoDBContext CreateBatchWrite not saving some items c#

I have created a method that saves data to AWS. It uses IDynamoDBContext and CreateBatchWrite method.

public class DynamoDbBaseRepository<T>
{
    private readonly IDynamoDBContext _context;

    public DynamoDbBaseRepository(IDynamoDBContext context)
    {
        _context = context;
    }

    public async Task<bool> AddMany(List<T> items)
    {
        BatchWrite<T> batch = _context.CreateBatchWrite<T>();
        batch.AddPutItems(items);
        await batch.ExecuteAsync();

        return true;
    }
}

When I try and save a request with approximately 3000 items a large number of items never get saved.

First I checked the AWS docs and found this:

Amazon DynamoDB limits the number of operations in a batch and the total size of the batch in a batch operation. For more information about the specific limits, see BatchWriteItem. If the API detects that your batch write request exceeded the allowed number of write requests or exceeded the maximum allowed HTTP payload size, it breaks the batch into several smaller batches. Additionally, if a response to a batch write returns unprocessed items, the API automatically sends another batch request with those unprocessed items.

From the above statement it seems that I have no need to loop over the request and split it in to batches as that complexity is abstracted away.

In the above statement it does refer the reader to here. But this documentation talks about BatchWriteItem which is not available from IDynamoDBContext. However it does show some data transfer limits which I assume will apply to my method too.

In short total data must be no more than Max 16MB, Max 25 items, Max per item 400KB.

But as previously mentioned I am expecting the below to handle that:

BatchWrite<T> batch = _context.CreateBatchWrite<T>();
batch.AddPutItems(items);

I thought that I was maybe running into an issue around ProvisionedThroughput so I increased the WriteCapacityUnits to 6, but this had no effect.

Am I using the wrong interface? Or should I be looking somewhere else in the table configuration?

Upvotes: 1

Views: 689

Answers (1)

tony09uk
tony09uk

Reputation: 2991

After finding no better alternatives I ignored the documentation and chunked the data into batches of 1000, this resolved the problem.

I assume there is a better solution but I was not able to find it. Here is the code I wrote:

List<Task> saveRequest = new List<Task>();
foreach(var chunk in items.ChunkBy(1000))
{
    saveRequest.Add(AddManyItems(chunk));
}

await Task.WhenAll(saveRequest);

The extension

public static class ListExtensions
{
    public static List<List<T>> ChunkBy<T>(this List<T> source, int chunkSize)
    {
        return source
            .Select((x, i) => new { Index = i, Value = x })
            .GroupBy(x => x.Index / chunkSize)
            .Select(x => x.Select(v => v.Value).ToList())
            .ToList();
    }
}

Upvotes: 1

Related Questions