CODEWITHSUNDEEP
amazon-web-servicesgoamazon-ec2bigdataamazon-dynamodb
WeCanBeFriends
WeCanBeFriends

Reputation: 691

PutItem into DynamoDB is slow

I have an EC2 instance which opens up a json file, reads each line and does a putItem operation into two tables.

Without the putItem operation, Golang parses a 67k line file in about 3 seconds.

With the putItem operation, it is processing 10k items every 5 minutes. The put operation into dynamodb is not being throttled. The WCUs and RCUs have been set accordingly. So Is there a reason that the putItem operation is holding up the code?

I assume that Golang is waiting for each put operation to succeed?

Still quite unsure, if anyone has done a mass insert into dynamodb using golang, then it would be helpful if you shedded some light on how you circumvented this.

Upvotes: 5

Views: 5081

Answers (1)

Zak
Zak

Reputation: 5898

The slowness comes from the fact that each insert has to do a full HTTP round trip to dynamo.

10k items in 5 mins is about 30ms per item, which is expected of an HTTP trip.

You can use a batch update, docs here

In the docs, you have BatchWriteItemInput, which takes a map[string][]*WriteRequest.

type BatchWriteItemInput struct {
    // ... trimmed

    RequestItems map[string][]*WriteRequest

    ...
}

The WriteRequest model is a little odd, because its used for both Delete and Put operations, just ignore the DeleteRequest *DeleteRequest field.

It's important to note that there are some limits on the batch operation:

  • You can't change the same item more than once in a batch request
  • You must have more than 2 and fewer than 25 items in a batch
  • Each item cannot be bigger than 400KB and the total batch cannot be bigger than 16MB.

Upvotes: 9

Related Questions