CODEWITHSUNDEEP
python-3.xamazon-web-servicesaws-lambdaboto3
Nick_Scott
Nick_Scott

Reputation: 107

Automate CSV File Creation Using Boto3 S3 and Lambda

I am doing a lab on the website LinuxAcademy.com The Course name is Automating AWS with Lambda, Python, and Boto3 and the specific lab I am having trouble with is Lecture: Importing CSV Files into DynamoDB.

In this lab we upload a .csv file into S3, an S3 event is generated in that specific bucket which then kicks off the Lambda script shown below:

import csv
import os
import tempfile

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Movies')
s3 = boto3.client('s3')


def lambda_handler(event, context):

    for record in event['Records']:
        source_bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']
        with tempfile.TemporaryDirectory() as tmpdir:
            download_path = os.path.join(tmpdir, key)
            s3.download_file(source_bucket, key, download_path)
            items = read_csv(download_file)

            with table.batch_writer() as batch:
                for item in items:
                    batch.put_item(Item=item)


def read_csv(file):
    items=[]
    with open(file) as csvfile:
        reader = csv.DictReader(csvfile)
        for row in reader:
            data = {}
            data['Meta'] = {}
            data['Year'] = int(row['Year'])
            data['Title'] = row['Title'] or none
            data['Meta']['Length'] = int(row['Length'] or 0)
            #data['Meta']['Length'] = int(row['Length'] or 0)
            data['Meta']['Subject'] = row['Subject'] or None
            data['Meta']['Actor'] = row['Actor'] or None
            data['Meta']['Actress'] = row['Actress'] or None
            data['Meta']['Director'] = row['Director'] or None
            data['Meta']['Popularity'] = row['Popularity'] or None
            data['Meta']['Awards'] = row['Awards'] == 'Yes'
            data['Meta']['Image'] = row['Image'] or None
            data['Meta'] = {k: v for k,
                            v in data['Meta'].items() if v is not None}

The .csv file upload to s3 does invoke the lambda function. I'm getting an error on line 20: items = read_csv(download_file)

Error from AWS CloudWatch:

[ERROR] NameError: name 'download_file' is not defined
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 20, in lambda_handler
    items = read_csv(download_file)

Upvotes: 0

Views: 531

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 270294

s3.download_file(source_bucket, key, download_path)
items = read_csv(download_file)

The first line is calling the download_file() method on the Amazon S3 client to download a file to the local disk.

The second line is calling the read_csv() function, passing a variable called download_file. However, no variable called download_file has been defined, which is why you are receiving the error.

In looking at the code, the read_csv() function is expecting the name of a file to open. This appears to be available in the download_path variable, which contains the local directory and key. Therefore, change it to:

items = read_csv(download_path)

Upvotes: 1

Related Questions