CODEWITHSUNDEEP
amazon-dynamodbaws-amplifyaws-appsyncdynamodb-queries
Marty Mitchener
Marty Mitchener

Reputation: 640

How to backfill new AppSync fields using AWS Amplify

I'm adding a sort field to one of my AppSync tables using GraphQL. The new schema looks like:

type MyTable
  @model
  @auth(rules: [{allow: owner}])
  @key(name: "BySortOrder", fields: ["sortOrder"], queryField: "tableBySortOrder")
{
  id: ID!
  name: String!
  sortOrder: Int
}

However, when retrieving a list using tableBySortOrder I get an empty list because the new field sortOrder is null.

My question is, how do I backfill this data in the DynamoDB table so that my existing users will not be disrupted by this new change? With a traditional database, I would run a SQL update: UPDATE MyTable SET sortOrder = #.

However, I'm new to NoSQL/AWS and couldn't find a way to do this except build a backfill script whenever a user logs into my app. That feels very hacky. What is the best practice for handling this type of scenario?

Upvotes: 0

Views: 1211

Answers (2)

Sunny
Sunny

Reputation: 176

Have you already created the new field in DDB? If yes, I think you should backfill it before making the client side change.

Write a script to iterate through and update the table. Options for this:

  • Java - Call updateItem to update the table if you have any integ tests running.
  • Bash - Use AWS CLI: aws dynamodb scan --table-name item_attributes --projection-expression "whatever" > /tmp/item_attributes_table.txt and then aws dynamodb update-item --table-name item_attributes --key. This is a dirty way.
  • Python - Same logic as above.

Upvotes: 1

Marty Mitchener
Marty Mitchener

Reputation: 640

Ended up using something similar to what Sunny suggested with a nodejs script:

const AWS = require('aws-sdk')

AWS.config.update({
  region: 'us-east-1'
})

// To confirm credentials are set
AWS.config.getCredentials(function (err) {
  if (err) console.log(err.stack)
  // credentials not loaded
  else {
    console.log('Access key:', AWS.config.credentials.accessKeyId)
    console.log('Secret access key:', AWS.config.credentials.secretAccessKey)
  }
})

const docClient = new AWS.DynamoDB.DocumentClient()

const table = 'your-table-dev'

const params = {
  TableName: table
}

const itemMap = new Map()

// Using scan to retrieve all rows
docClient.scan(params, function (err, data) {
  if (err) {
    console.error('Unable to query. Error:', JSON.stringify(err, null, 2))
  } else {
    console.log('Query succeeded.')
    data.Items.forEach(item => {
      if (itemMap.has(item.owner)) {
        itemMap.set(item.owner, [...itemMap.get(item.owner), item])
      } else {
        itemMap.set(item.owner, [item])
      }
    })

    itemMap.forEach(ownerConnections => {
      ownerConnections.forEach((connection, index) => {
        connection.sortOrder = index
        update(connection)
      })
    })
  }
})

function update(connection) {
  const params = {
    TableName: table,
    Key: {
      'id': connection.id
    },
    UpdateExpression: 'set sortOrder = :s',
    ExpressionAttributeValues: {
      ':s': connection.sortOrder,
    },
    ReturnValues: 'UPDATED_NEW'
  };

  console.log('Updating the item...');
  docClient.update(params, function (err, data) {
    if (err) {
      console.error('Unable to update item. Error JSON:', JSON.stringify(err, null, 2));
    } else {
      console.log('UpdateItem succeeded:', JSON.stringify(data, null, 2));
    }
  });
}

Upvotes: 0

Related Questions