CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3aws-lambdaamazon-dynamodb
Abhishek
Abhishek

Reputation: 3

AWS lambda function to write csv data to an Amazon DynamoDB table

Below is the code to trigger a lambda function when a csv file is uploaded to an Amazon S3 bucket and write data to an Amazon DynamoDB table. I am getting error "expected str, bytes or os.PathLike object, not dict".

Could you please suggest where is the mistake I did.

import boto3
import csv

s3_client = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
#table = dynamodb.Table('batch_data')

def csv_write(table_name, rows):
    table = dynamodb.Table(table_name)

    with table.csv_write() as batch:
        for row in rows:
            batch.put_item(Item=row)
    return True

def read_csv(csv_file, list):
    rows = csv.DictReader(open(csv_file))

    for row in rows:
        list.append(row)

def lambda_handler(event, context):
    try:

        bucket = event['Records'][0]['s3']['bucket']['name']
        csv_file_name = event['Records'][0]['s3']['object']['key']
        response = s3_client.get_object(Bucket=bucket, Key= csv_file_name)

        table_name = 'batch_data'
        items = []

        read_csv(response, items)
        status = csv_write(table_name, items)

        if(status):
            print('Data saved')
        else:
            print('Error in saving data...')
    except Exception as err:
        print (err)
    ```

Upvotes: 0

Views: 1396

Answers (2)

John Rotenstein
John Rotenstein

Reputation: 270224

This line:

response = s3_client.get_object(Bucket=bucket, Key= csv_file_name)

returns an object like this:

{
    'Body': StreamingBody(),
    'LastModified': datetime(2015, 1, 1),
    'ContentLength': 123,
    ...
}

This line:

read_csv(response, items)

is passing that response to this function as csv_file:

def read_csv(csv_file, list):
    rows = csv.DictReader(open(csv_file))
    ...

However, the open() command doesn't know how to interpret the response object from S3. That is why it is returning the error:

expected str, bytes or os.PathLike object, not dict

Based on information from How do I read a csv stored in S3 with csv.DictReader?, I got this code working:

import boto3
import csv

def lambda_handler(event, context):
    
    s3_client = boto3.client('s3')
    
    bucket = event['Records'][0]['s3']['bucket']['name']
    csv_file_name = event['Records'][0]['s3']['object']['key']
    
    response = s3_client.get_object(Bucket=bucket, Key= csv_file_name)

    lines = response['Body'].read().decode('utf-8').split()
    for row in csv.DictReader(lines):
        print(row)

You can use that method to read the CSV from the StreamingBody that is returned from get_object().

Upvotes: 1

bumpbump
bumpbump

Reputation: 804

likely your bucket or csv_file_name is inadvertently a dictionary, i.e. you didn't exhaust the hierarchy of event

Upvotes: 0

Related Questions