CODEWITHSUNDEEP
pythonamazon-s3aws-lambda
aysh
aysh

Reputation: 599

How to insert from csv to dynamodb

Dynamo db have one table employees primary key as id

data.csv is below which uploaded in the csvdynamo bucket

bucket_name = csvdynamo

id,name,co
20,AB,PC
21,CD,PC
22,EF,MC
23,GH,MC

Need to insert above csv into dynamodb

psuedo code

for emp in employees:
    emp_data= emp.split(',')
    print (emp_data)

    try:
        table.put_item(
            Item = {
                "emp_id": int(emp_data[0]),
                "Name": emp_data[1],
                "Company": emp_data[2]
            }
        )
    except Exception as e:
          pass

Upvotes: 2

Views: 10738

Answers (3)

Thunderwrathy
Thunderwrathy

Reputation: 81

Here is a script for those who just want to import a csv file that is locally on their computer to a DynamoDB table. (I just took the script from @Marcin and modified it a little bit, leaving out the S3 bucket code and making it more general.)

import boto3


csv_file_path = "your-csv-file.csv"
table_name = "your-dynamodb-table-name"
db_table = boto3.resource('dynamodb').Table(table_name)
line_seperator = ';'


def save_to_dynamodb(column_names, values):
    item = dict()

    for idx, column_name in enumerate(column_names):
        item[column_name.lower()] = values[idx]

    return db_table.put_item(
        Item=item
    )


def handler():

    with open(csv_file_path, 'r', encoding='utf-8-sig') as f:

        column_names = next(f).strip("\n").split(line_seperator)

        for line in f:
            values = line.strip("\n").split(line_seperator)

            result = save_to_dynamodb(column_names, values)

            print(result)

    print("FINISHED IMPORT")


handler()

Upvotes: 0

Marcin
Marcin

Reputation: 238199

Here is an example of a lambda function which works, as I verified it using my own function, csv file and dynamodb. I think the code is self-explanatory. It should be a good start towards your end use-case.

          import boto3
          import json
          import os

          bucket_name = os.environ['BUCKET_NAME']
          csv_key = os.environ['CSV_KEY_NAME'] # csvdynamo.csv
          table_name = os.environ['DDB_TABLE_NAME']
          
          # temprorary file to store csv downloaded from s3
          tmp_csv_file = '/tmp/' + csv_key

          s3 = boto3.resource('s3')
          db_table = boto3.resource('dynamodb').Table(table_name)

          def save_to_dynamodb(id, name, co):

            return db_table.put_item(
                Item={
                  'emp_id': int(id),
                  'Name': name,
                  'Company': co
                })

          def lambda_handler(event, context):

              s3.meta.client.download_file(
                            bucket_name, 
                            csv_key, 
                            tmp_csv_file)

              with open(tmp_csv_file, 'r') as f:

                next(f) # skip header

                for line in f:

                  id, name, co = line.rstrip().split(',')

                  result = save_to_dynamodb(id, name, co)

                  print(result)

              return {'statusCode': 200}

Upvotes: 6

John Rotenstein
John Rotenstein

Reputation: 269340

DynamoDB needs to be told the type of field being inserted.

From put_item(), the format for Item is:

    Item={
        'string': {
            'S': 'string',
            'N': 'string',
            'B': b'bytes',
            'SS': [
                'string',
            ],
            'NS': [
                'string',
            ],
            'BS': [
                b'bytes',
            ],
            'M': {
                'string': {'... recursive ...'}
            },
            'L': [
                {'... recursive ...'},
            ],
            'NULL': True|False,
            'BOOL': True|False
        }
    },

Therefore, you would need something like:

        table.put_item(
            Item = {
                "emp_id":  {'N': emp_data[0]},
                "Name":    {'S': emp_data[1]},
                "Company": {'S': emp_data[2]}
            }
        )

Upvotes: 1

Related Questions