CODEWITHSUNDEEP
node.jsamazon-web-servicesamazon-s3aws-lambdaamazon-dynamodb
Luke Browne
Luke Browne

Reputation: 131

Converting JSON Data from S3 upload, and using Lambda function to push to DynamoDB

I've been working on an assignment recently and I feel like I'm very close to solving the problem I'm having, but I just can't seem to find anything that would help online. As the title states, I've got some JSON data being uploaded from a webpage into an S3 bucket. When a new S3 item is created, I want to take that data and store it in a DynamoDB table.

I'm using a Lambda function and testing with some data I've already stored in my S3 bucket. I've got the data in its key-value pairs in my console.logs but I just can't work out why it isn't actually storing the data.

Example of key-value pairs in my Lambda function test log

On the left I have the data broken down into its key-value pair, i.e. "artist": "Elvis Presley", using JSON.parse(JSON.stringify(data)).

What I'm wondering, is how to push this data into the table.

var params = {
              Item: JSON.parse(JSON.stringify(data)),
              ReturnConsumedCapacity: "TOTAL",
              TableName: "s3-to-dynamo-s00187306"
             };
             dynamo.putItem(params, dynamoResultCallback);

The above code is what I've been trying to use but it's giving me a timeout error. If I bump up the allowed time then I receive a different error relating to a missing partition key in the item, even though my partition key matches with one of the key values in every item.

Really stumped here, any advice is appreciated, thanks in advance.

[edit]

So I used what someone suggested below, the dynamo-db converter, and have some logs which provide some insight into what's going on.

enter image description here

I've now got the data in the correct format for dynamo-db, and each item is parsed correctly as far as I can tell.

As for what dynamo represents, I'm not 100% so I'm going to add a screenshot of its declaration at the top of my code. I think it's the doc client?

[edit 2]

So my "_class" values are all the exact same, might try changing the partition key to title instead? (nevermind this didn't work)

enter image description here

Upvotes: 0

Views: 1102

Answers (1)

Youcef LAIDANI
Youcef LAIDANI

Reputation: 59986

JSON.stringify(data) return a json format that not match with Dynamodb format, Dynamodb are waiting a format like this:

Item: {
    'CUSTOMER_ID' : {N: '001'},
    'CUSTOMER_NAME' : {S: 'Richard Roe'}
}

As you see, the syntax is not the same, I think you need to use another library, maybe dynamo-converters, or look at NodeJs Aws SDK maybe there is a method that can do this.

Upvotes: 1

Related Questions