CODEWITHSUNDEEP
amazon-web-serviceskubernetesamazon-dynamodblocalstackamazon-dynamodb-local
Vamshi Krishna
Vamshi Krishna

Reputation: 99

Localstack DynamoDB : PutItem operation exceeded maximum number of attempts

I set up a DynamoDB table on a Kubernetes pod using LocalStack, which is exposed via a Kubernetes service. I'm trying to make a PutItem request to this table using a service configured with the following DynamoDB client setup:

var client *dynamodb.Client
var err error

httpClient := awshttp.NewBuildableClient().WithTransportOptions(
    func(tr *http.Transport) {
        tr.IdleConnTimeout = cfg.DDB.IdleConnTimeout
    }).WithTimeout(cfg.DDB.ReqTimeout)

ddbClientConfig, err = awsconfig.LoadDefaultConfig(
    context.Background(),
    awsconfig.WithRetryer(func() aws.Retryer {
        return retry.NewStandard(func(options *retry.StandardOptions) {
            options.MaxAttempts = cfg.DDB.MaxRetryCount
            options.MaxBackoff = cfg.DDB.MaxBackoffDelay
        })
    }),
    awsconfig.WithHTTPClient(httpClient),
)

ddbClient = dynamodb.NewFromConfig(clientCfg, func(o *dynamodb.Options) {
    o.BaseEndpoint = "http://<localstack-service-name>.<kubernetes-namespace-name>.svc.cluster.local:91"
})

However, when making a PutItem request using this DynamoDB client, I encounter the following error:

operation error DynamoDB: PutItem, exceeded maximum number of attempts, 3, https response error StatusCode: 503, RequestID: , api error UnknownError: UnknownError

I have explored several solutions, including:

  1. Verifying network connectivity between the pods.
  2. Increasing the memory and CPU allocations for the LocalStack pod.
  3. Upgrading the aws-sdk-go-v2/dynamodb version used in the client application.

Despite making these changes, the issue persists. The table was set up in LocalStack with the following command:

awslocal dynamodb create-table \
 --endpoint-url http://localhost:4566 \ # The Kubernetes service directs this localhost 4566 port to expose port 91
 --region us-east-1 \
 --table-name transaction-data \
 --attribute-definitions \
     AttributeName=key1,AttributeType=S \
     AttributeName=key2,AttributeType=S \
 --key-schema AttributeName=key1,KeyType=HASH \
              AttributeName=key2,KeyType=RANGE \
 --billing-mode PAY_PER_REQUEST

Interestingly, I am able to successfully execute a curl request from the client pod to the LocalStack pod to put an item into the table. However, the PutItem request made from the client application using the Go AWS SDK results in the error mentioned above.

I also attempted using the --sharedDb parameter when starting the LocalStack container in my Dockerfile, and also setting it in my kubernetes deployment help chart but without success:

FROM localstack/localstack:latest

# Set environment variables for test AWS credentials
ENV AWS_ACCESS_KEY_ID=test_key_id
ENV AWS_SECRET_ACCESS_KEY=test_key
ENV AWS_REGION=us-east-1


COPY /execute-scripts/scripts/ /etc/localstack/init/ready.d/

RUN chmod +x /etc/localstack/init/ready.d/executeScript.sh

# Starting localstack process with --shareDb parameter
CMD ["localstack", "start", "--sharedDb"]

Can anyone help identify potential reasons for this error and suggest solutions?

UPDATE :

I also am passing in the --sharedDb flag to localstack container as env variables my kubernetes deployment helm chart, but I still see the DDB service in the localstack container is setting --sharedDb to false. Am I using --sharedDb parameter incorrectly?

            env:
              - name: JAVA_OPTS
                value: "-Xms512m -Xmx512m"
              - name: SERVICES
                value: "dynamodb,s3,sqs"
              - name: DEBUG
                value: "1"
              - name: LOG_LEVEL
                value: "TRACE"
              - name: LOG_FORMAT
                value: "json"
            args: ["start","-sharedDb","--async-response-enabled", "true", "--async-response-threads", "30", "--no-request-journal", "--jetty-acceptor-threads","10"]

localstack container logs

Initializing DynamoDB Local with the following configuration:
Port:   54919
InMemory:   false
Version:    2.5.4
SharedDb:   false

Update 2

I updated my application's DDB client to use the same access key values which the localstack docker container is using. This also did not help with the issue

client = dynamodb.NewFromConfig(clientCfg, func(o *dynamodb.Options) {
                o.BaseEndpoint = &cfg.LocalStackEndpoint
                o.Credentials = awsCredentials.NewStaticCredentialsProvider("test_key_id", "test_key", "")
            })

Upvotes: 0

Views: 44

Answers (1)

Leeroy Hannigan
Leeroy Hannigan

Reputation: 19873

When you create DynamoDB local pass the table -sharedDb parameter which you can also set as an env variable for local stack.

Because you create the tables using CLI which has different access keys it creates a different database file to the one from your code. shareDb uses a shared database file so you can access it from any local code.

If you use the -sharedDb option, DynamoDB creates a single database file named shared-local-instance.db. Every program that connects to DynamoDB accesses this file. If you delete the file, you lose any data that you have stored in it.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.UsageNotes.html

In localstack that translates to the following:

DYNAMODB_SHARE_DB=1

CMD ["DYNAMODB_SHARE_DB=1","localstack", "start"]

Upvotes: 1

Related Questions