CODEWITHSUNDEEP
c#.netamazon-dynamodb
Pawan
Pawan

Reputation: 2014

BatchWriteItem using DynamoDB and AWSSDK.net

I am currently having issues when calling a BatchWriteItem Request using DynamoDB. I get the following error

"Unable to save data to DB. The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API"

Configuration:

ThroughPut : Read = 5 ; Write = 5;

This is my scenario: I need to perform 2 sets of Data Write request one after the other to the same table - using aggregated Data (932KB: split into chunks of 63KB which is approximately 15 chunks) - Non-Aggregated Data (940 KB: split into chunks of 63KB which is approximately 15 chunks )

Before doing the write i delete all the items in the dynamo-db table based on the hash key and range key values (for aggregate Data : 16 items are deleted).

Below is the code:

public void PutItems(string tableName, string id, string message, bool aggregated)
    {
         if (aggregated)
             id = "a_" + id;
         else
             id = "r_" + id;
        List<WriteRequest> writeRequests = new List<WriteRequest>();
        List<WriteRequest> DeleteWriteRequest = new List<WriteRequest>();
        var itemsToDelete = GetItemsToDelete(tableName, id);
        if (itemsToDelete.Count > 0)
        {
            foreach (Tuple<string, int> item in itemsToDelete)
            {

                DeleteRequest deleteRequest = new DeleteRequest
                {
                    Key = new Key
                    {
                        HashKeyElement = new AttributeValue { S = item.Item1 },
                        RangeKeyElement = new AttributeValue { N = item.Item2.ToString() }
                    }
                };
                WriteRequest deleteReq = new WriteRequest();
                deleteReq.DeleteRequest = deleteRequest;
                DeleteWriteRequest.Add(deleteReq);
            }
            logger.Info(this.GetType().ToString(), string.Format(".....Deleting Old Data..........."));
            foreach (IEnumerable<WriteRequest> ls in LinqExtensions.Partition(DeleteWriteRequest, SplitLevel))
            {
                BatchWriteItemRequest delWriteRequest = new BatchWriteItemRequest();
                delWriteRequest.WithRequestItems(new KeyValuePair<string, List<WriteRequest>>(tableName, ls.ToList()));

                CallBatchWriteTillCompletion(delWriteRequest);
            }
            logger.Info(this.GetType().ToString(), string.Format(".....Delete Complete!..........")); 
        }

        int MaxLength = 64512; //64KB = 65536 Bytes ; 60KB = 61440 Bytes
        logger.Info(this.GetType().ToString(), string.Format("Message Size : {0}", message.Length)); 
        var str = SplitToChunks(message, MaxLength).ToList();
        for (int i = 0; i < str.Count; i++)
        {



            PutRequest putRequest = new PutRequest
            {
                Item = new Dictionary<string, AttributeValue>()
                    {
                        {"Received", new AttributeValue {S = id}},
                        {"SequenceNum" , new AttributeValue {N = i.ToString()}},
                        {"Message", new AttributeValue {S = str[i]}}
                    }
            };
            WriteRequest request = new WriteRequest();
            request.PutRequest = putRequest;
            writeRequests.Add(request);
        }

        logger.Info(this.GetType().ToString(), string.Format(".....Writing Data..........."));
        foreach (IEnumerable<WriteRequest> ls in LinqExtensions.Partition(writeRequests, SplitLevel))
        {
                System.Threading.Thread.Sleep(1000);
                BatchWriteItemRequest writeRequest = new BatchWriteItemRequest();
                writeRequest.WithRequestItems(new KeyValuePair<string, List<WriteRequest>>(tableName, ls.ToList()));
                CallBatchWriteTillCompletion(writeRequest);


        }
        logger.Info(this.GetType().ToString(), string.Format(".....Write Complete!.........."));
    }


    private void CallBatchWriteTillCompletion(BatchWriteItemRequest request)
    {
        BatchWriteItemResponse response;

        int callCount = 0;
        do
        {
            if (callCount > 0)
            {
                System.Threading.Thread.Sleep(1000);
            }
            logger.Info(this.GetType().ToString(), string.Format("Making Request"));
            response = Instance.Client.BatchWriteItem(request);
            callCount++;

            // Check the response.
            var result = response.BatchWriteItemResult;
            var responses = result.Responses;
            var unprocessed = result.UnprocessedItems;

            logger.Info(this.GetType().ToString(), string.Format("Response"));
            foreach (var resp in responses)
            {

                logger.Info(this.GetType().ToString(), string.Format("{0} - {1}", resp.Key, resp.Value.ConsumedCapacityUnits));
            }

            logger.Info(this.GetType().ToString(), string.Format("Unprocessed"));
            foreach (var unp in unprocessed)
            {

                logger.Info(this.GetType().ToString(), string.Format("{0} - {1}", unp.Key, unp.Value.Count));

            }

            // For the next iteration, the request will have unprocessed items.
            request.RequestItems = unprocessed;

        } while (response.BatchWriteItemResult.UnprocessedItems.Count > 0);

        logger.Info(this.GetType().ToString(), string.Format("Total # of batch write API calls made: {0}", callCount));

    }



    //This method is use to split a string message of smaller chunks of 64KB.
    private static IEnumerable<String> SplitToChunks(String str, int maxLength)
    {
            for (int index = 0; index < str.Length; index += maxLength)
            {
                yield return str.Substring(index, Math.Min(maxLength, str.Length - index));
            }          
    }

static class LinqExtensions {

    public static IEnumerable<IEnumerable<T>> Partition<T>(this IEnumerable<T> source, int size)
    {
        T[] array = null;
        int count = 0;
        foreach (T item in source)
        {
            if (array == null)
            {
                array = new T[size];
            }
            array[count] = item;
            count++;
            if (count == size)
            {
                yield return new ReadOnlyCollection<T>(array);
                array = null;
                count = 0;
            }
        }
        if (array != null)
        {
            Array.Resize(ref array, count);
            yield return new ReadOnlyCollection<T>(array);
        }
    }

}

Can you tell me what i am doing wrong here. I have added delays of 1 sec at multiple places so that the through put never exceeds. I have also attached a text file with log which has the ConsumedCapacityUnits.

2013-03-06 21:20:21.6093 INFO     
2013-03-06 21:20:21.7031 INFO     .........Writing Aggregated Messages to DynamoDB ...
2013-03-06 21:20:21.7343 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:20:22.2500 INFO     Working mode: MASTER.
2013-03-06 21:20:22.3750 INFO     Heartbeat message '3/6/2013 9:20:22 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:20:22.9062 INFO     .....Deleting Old Data...........
2013-03-06 21:20:22.9531 INFO     Making Request
2013-03-06 21:20:23.1250 INFO     Response
2013-03-06 21:20:23.1562 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 320
2013-03-06 21:20:23.2031 INFO     Unprocessed
2013-03-06 21:20:23.2500 INFO     Total # of batch write API calls made: 1
2013-03-06 21:20:23.2812 INFO     Making Request
2013-03-06 21:20:23.4218 INFO     Response
2013-03-06 21:20:23.5468 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 320
2013-03-06 21:20:23.5937 INFO     Unprocessed
2013-03-06 21:20:23.6406 INFO     Total # of batch write API calls made: 1
2013-03-06 21:20:23.6718 INFO     Making Request
2013-03-06 21:20:26.1406 INFO     Response
2013-03-06 21:20:26.2031 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 320
2013-03-06 21:20:26.2500 INFO     Unprocessed
2013-03-06 21:20:26.2968 INFO     Total # of batch write API calls made: 1
2013-03-06 21:20:26.3593 INFO     Making Request
2013-03-06 21:20:28.6875 INFO     Response
2013-03-06 21:20:28.7187 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 54
2013-03-06 21:20:28.7500 INFO     Unprocessed
2013-03-06 21:20:28.7968 INFO     Total # of batch write API calls made: 1
2013-03-06 21:20:28.8750 INFO     .....Delete Complete!..........
2013-03-06 21:20:28.9218 INFO     Message Size : 1022580
2013-03-06 21:20:28.9687 INFO     .....Writing Data...........
2013-03-06 21:20:30.0000 INFO     Making Request
2013-03-06 21:20:33.2812 INFO     Response
2013-03-06 21:20:33.3125 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 128
2013-03-06 21:20:33.3437 INFO     Unprocessed
2013-03-06 21:20:33.3906 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 3
2013-03-06 21:20:34.4218 INFO     Making Request
2013-03-06 21:20:36.2343 INFO     Response
2013-03-06 21:20:36.2656 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 128
2013-03-06 21:20:36.2968 INFO     Unprocessed
2013-03-06 21:20:36.3281 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 1
2013-03-06 21:20:36.6093 INFO     
2013-03-06 21:20:36.6406 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:20:37.0156 INFO     Working mode: MASTER.
2013-03-06 21:20:37.0781 INFO     Heartbeat message '3/6/2013 9:20:37 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:20:37.3750 INFO     Making Request
2013-03-06 21:20:38.0781 INFO     Response
2013-03-06 21:20:38.1093 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:20:38.1406 INFO     Unprocessed
2013-03-06 21:20:38.1718 INFO     Total # of batch write API calls made: 3
2013-03-06 21:20:39.2187 INFO     Making Request
2013-03-06 21:20:42.2812 INFO     Response
2013-03-06 21:20:42.3125 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 128
2013-03-06 21:20:42.3437 INFO     Unprocessed
2013-03-06 21:20:42.3906 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 3
2013-03-06 21:20:43.4218 INFO     Making Request
2013-03-06 21:20:45.2187 INFO     Response
2013-03-06 21:20:45.2500 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 128
2013-03-06 21:20:45.2968 INFO     Unprocessed
2013-03-06 21:20:45.3281 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 1
2013-03-06 21:20:46.3593 INFO     Making Request
2013-03-06 21:20:47.0156 INFO     Response
2013-03-06 21:20:47.0625 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:20:47.0937 INFO     Unprocessed
2013-03-06 21:20:47.1250 INFO     Total # of batch write API calls made: 3
2013-03-06 21:20:48.1718 INFO     Making Request
2013-03-06 21:20:51.2968 INFO     Response
2013-03-06 21:20:51.3437 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 128
2013-03-06 21:20:51.3906 INFO     Unprocessed
2013-03-06 21:20:51.4218 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 3
2013-03-06 21:20:51.6093 INFO     
2013-03-06 21:20:51.6406 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:20:52.0000 INFO     Working mode: MASTER.
2013-03-06 21:20:52.0781 INFO     Heartbeat message '3/6/2013 9:20:52 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:20:52.4531 INFO     Making Request
2013-03-06 21:20:54.2656 INFO     Response
2013-03-06 21:20:54.3125 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 128
2013-03-06 21:20:54.3437 INFO     Unprocessed
2013-03-06 21:20:54.3750 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 1
2013-03-06 21:20:55.6718 INFO     Making Request
2013-03-06 21:20:56.3281 INFO     Response
2013-03-06 21:20:56.3750 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:20:56.4375 INFO     Unprocessed
2013-03-06 21:20:56.5312 INFO     Total # of batch write API calls made: 3
2013-03-06 21:20:57.5937 INFO     Making Request
2013-03-06 21:20:58.1875 INFO     Response
2013-03-06 21:20:58.2343 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 54
2013-03-06 21:20:58.3281 INFO     Unprocessed
2013-03-06 21:20:58.5625 INFO     Total # of batch write API calls made: 1
2013-03-06 21:20:58.5937 INFO     .....Write Complete!..........
2013-03-06 21:20:58.6875 INFO     .........Writing Complete! ...
2013-03-06 21:20:58.7187 INFO     .........Writing Raw Messages to DynamoDB ...
2013-03-06 21:21:00.8437 INFO     .....Deleting Old Data...........
2013-03-06 21:21:00.8906 INFO     Making Request
2013-03-06 21:21:06.0000 INFO     New Message Received
2013-03-06 21:21:06.6093 INFO     
2013-03-06 21:21:06.6562 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:21:07.0468 INFO     Working mode: MASTER.
2013-03-06 21:21:07.1250 INFO     Heartbeat message '3/6/2013 9:21:07 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:21:08.2343 INFO     Response
2013-03-06 21:21:08.2812 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 39
2013-03-06 21:21:08.3281 INFO     Unprocessed
2013-03-06 21:21:08.4062 INFO     Total # of batch write API calls made: 1
2013-03-06 21:21:08.4843 INFO     .....Delete Complete!..........
2013-03-06 21:21:08.5156 INFO     Message Size : 1006556
2013-03-06 21:21:08.5625 INFO     .....Writing Data...........
2013-03-06 21:21:09.6093 INFO     Making Request
2013-03-06 21:21:15.7187 INFO     Response
2013-03-06 21:21:15.7656 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:21:15.7968 INFO     Unprocessed
2013-03-06 21:21:15.8437 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 4
2013-03-06 21:21:16.8750 INFO     Making Request
2013-03-06 21:21:21.6093 INFO     
2013-03-06 21:21:21.6562 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:21:23.4687 INFO     Working mode: MASTER.
2013-03-06 21:21:24.1406 INFO     Heartbeat message '3/6/2013 9:21:23 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:21:27.3593 INFO     Response
2013-03-06 21:21:27.4062 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:21:27.4531 INFO     Unprocessed
2013-03-06 21:21:27.5000 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 3
2013-03-06 21:21:28.5312 INFO     Making Request
2013-03-06 21:21:36.6093 INFO     
2013-03-06 21:21:36.9687 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:21:38.1718 INFO     Working mode: MASTER.
2013-03-06 21:21:38.2343 INFO     Heartbeat message '3/6/2013 9:21:38 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:21:40.2031 INFO     Response
2013-03-06 21:21:40.2500 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:21:40.3281 INFO     Unprocessed
2013-03-06 21:21:40.3593 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 2
2013-03-06 21:21:41.3906 INFO     Making Request
2013-03-06 21:21:51.6093 INFO     
2013-03-06 21:21:51.7500 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:21:52.1718 INFO     Working mode: MASTER.
2013-03-06 21:21:52.2500 INFO     Heartbeat message '3/6/2013 9:21:52 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:21:54.0937 INFO     Response
2013-03-06 21:21:54.1562 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:21:54.1875 INFO     Unprocessed
2013-03-06 21:21:54.2343 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 1
2013-03-06 21:21:55.2812 INFO     Making Request
2013-03-06 21:22:06.2031 INFO     New Message Received
2013-03-06 21:22:06.6093 INFO     
2013-03-06 21:22:06.6406 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:22:07.0312 INFO     Working mode: MASTER.
2013-03-06 21:22:07.5781 INFO     Response
2013-03-06 21:22:07.6406 INFO     Heartbeat message '3/6/2013 9:22:07 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:22:07.7812 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:22:07.8437 INFO     Unprocessed
2013-03-06 21:22:07.9375 INFO     Total # of batch write API calls made: 5
2013-03-06 21:22:08.9687 INFO     Making Request
2013-03-06 21:22:18.0156 INFO     Response
2013-03-06 21:22:18.0781 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:22:18.1250 INFO     Unprocessed
2013-03-06 21:22:18.1562 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 4
2013-03-06 21:22:19.2031 INFO     Making Request
2013-03-06 21:22:21.6093 INFO     
2013-03-06 21:22:21.6875 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:22:21.7031 INFO     Response
2013-03-06 21:22:21.7656 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:22:21.8281 INFO     Unprocessed
2013-03-06 21:22:21.8750 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 3
2013-03-06 21:22:22.0937 INFO     Working mode: MASTER.
2013-03-06 21:22:22.1875 INFO     Heartbeat message '3/6/2013 9:22:22 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:22:22.9375 INFO     Making Request
2013-03-06 21:22:34.9687 INFO     Response
2013-03-06 21:22:35.0156 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:22:35.0468 INFO     Unprocessed
2013-03-06 21:22:35.0781 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 2
2013-03-06 21:22:36.1093 INFO     Making Request
2013-03-06 21:22:36.6093 INFO     
2013-03-06 21:22:36.7656 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:22:38.2968 INFO     Working mode: MASTER.
2013-03-06 21:22:38.5468 INFO     Heartbeat message '3/6/2013 9:22:38 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:22:48.7968 INFO     Response
2013-03-06 21:22:48.8750 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:22:48.9062 INFO     Unprocessed
2013-03-06 21:22:48.9375 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 1
2013-03-06 21:22:50.0000 INFO     Making Request
2013-03-06 21:22:51.6093 INFO     
2013-03-06 21:22:51.8281 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:22:52.8125 INFO     Working mode: MASTER.
2013-03-06 21:22:52.8906 INFO     Heartbeat message '3/6/2013 9:22:52 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:23:02.2968 INFO     Response
2013-03-06 21:23:02.3281 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:23:02.3593 INFO     Unprocessed
2013-03-06 21:23:02.3906 INFO     Total # of batch write API calls made: 5
2013-03-06 21:23:03.4375 INFO     Making Request
2013-03-06 21:23:06.6093 INFO     
2013-03-06 21:23:06.7812 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:23:07.5000 INFO     New Message Received
2013-03-06 21:23:08.8593 INFO     Working mode: MASTER.
2013-03-06 21:23:09.3281 INFO     Heartbeat message '3/6/2013 9:23:09 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:23:15.3906 INFO     Response
2013-03-06 21:23:15.4375 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:23:15.4843 INFO     Unprocessed
2013-03-06 21:23:15.5312 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 4
2013-03-06 21:23:16.5625 INFO     Making Request
2013-03-06 21:23:21.6093 INFO     
2013-03-06 21:23:21.8593 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:23:23.7031 INFO     Working mode: MASTER.
2013-03-06 21:23:24.0468 INFO     Heartbeat message '3/6/2013 9:23:23 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:23:28.7812 INFO     Response
2013-03-06 21:23:28.8125 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 64
2013-03-06 21:23:28.8750 INFO     Unprocessed
2013-03-06 21:23:29.1718 INFO     Key: TrafficConditionsMessages_dev - Unprocessed Value Count: 3
2013-03-06 21:23:30.4375 INFO     Making Request
2013-03-06 21:23:31.6093 INFO     Run task Called......
2013-03-06 21:23:31.9375 INFO     Processing Read data Method
2013-03-06 21:23:32.7187 INFO     Saving data to DB...
2013-03-06 21:23:33.0156 INFO     .........Writing Aggregated Messages to DynamoDB ...
2013-03-06 21:23:36.8125 INFO     
2013-03-06 21:23:37.0156 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:23:37.4531 INFO     .....Deleting Old Data...........
2013-03-06 21:23:37.7500 INFO     Making Request
2013-03-06 21:23:38.5156 INFO     Working mode: MASTER.
2013-03-06 21:23:38.8593 INFO     Heartbeat message '3/6/2013 9:23:38 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:23:41.6875 INFO     Response
2013-03-06 21:23:41.9531 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 320
2013-03-06 21:23:42.2500 INFO     Unprocessed
2013-03-06 21:23:42.3593 INFO     Total # of batch write API calls made: 1
2013-03-06 21:23:42.4218 INFO     Making Request
2013-03-06 21:23:50.5468 ERROR    Unable to save data to DB. The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API
2013-03-06 21:23:50.5781 INFO     .....Save to Database Complete!..... 
2013-03-06 21:23:50.7812 INFO     Run task Complete......
2013-03-06 21:23:51.6093 INFO     
2013-03-06 21:23:51.6406 INFO     Heartbeat event fired. Processing heartbeat ...
2013-03-06 21:23:53.1718 INFO     Working mode: MASTER.
2013-03-06 21:23:53.4531 INFO     Heartbeat message '3/6/2013 9:23:53 PM|OAK-VENUGOPALP' written to SQS queue 'HeartbeatDataReceiver_dev'.
2013-03-06 21:23:53.6093 ERROR    Unable to save data to DB. The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API
2013-03-06 21:23:53.6562 INFO     .....Save to Database Complete!..... 
2013-03-06 21:23:53.8281 INFO     Run task Complete......

Upvotes: 1

Views: 3229

Answers (1)

Cory Kendall
Cory Kendall

Reputation: 7324

This number translates (roughly) to KB per second: ThroughPut : Write = 5;

If you are trying to write an item of 63KB, you will need to write it, and then wait ~13 seconds before attempting to write again. You have provisioned 5 KB per second, and then used 63 KB in a single burst. Over the next 13 seconds, any request you make to write to the table could be Throttled.

Judging by the timestamps, the client is invisibly handling the ProvisionedThroughputExceeded errors and retrying; for example, here you see 10 seconds pass during a request:

2013-03-06 21:22:08.9687 INFO     Making Request
2013-03-06 21:22:18.0156 INFO     Response

Eventually, the table was hit with a big request after a bunch of smaller ones:

2013-03-06 21:23:41.9531 INFO     Key: TrafficConditionsMessages_dev - Consumed Capacity Units: 320

and the next request took so long that your client decided it was time to surface the error to you:

2013-03-06 21:23:42.4218 INFO     Making Request
2013-03-06 21:23:50.5468 ERROR    Unable to save data to DB. The ...

The solution is either to up your provisioned throughput, sleep for longer based on the returned ConsumedCapacity, or write less data.

Upvotes: 1

Related Questions