CODEWITHSUNDEEP
pythonpostgresqlamazon-s3aws-lambdaamazon-rds
wendybear
wendybear

Reputation: 127

CSV-File from AWS S3 into PostgreSQL Amazon RDS using python

Status:

  1. I have created new tables in PostgreSQL-Database on Amazon RDS
  2. I have uploaded a csv-file into Bucket on Amazon S3
  3. via lambda function I have connected with Amazon S3 Buckets and Amazon RDS

  4. I can read csv-file via the following code

    import csv, io, boto3

s3 = boto3.resource('s3')
client = boto3.client('s3',aws_access_key_id=Access_Key,aws_secret_access_key=Secret_Access_Key)
buf = io.BytesIO()
s3.Object('bucketname','filename.csv').download_fileobj(buf)

buf.seek(0)
while True:
    line =  buf.readlines(1)
    print(line)

Problem:

I can't import necessary python libraries e.g. psycopg2, openpyxl etc. when I tried to import psycopg2

    import psycopg2

I got the error info:

    Unable to import module 'myfilemane': No module named 'psycopg2._psycopg'

at first I have not imported the module "psycopg2._psycopg" but "psycopg2". I don't know where is the suffix '_psycopg' from

secondly I followed all the steps in the documentation: https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html (1. create a directory. 2. Save all of your Python source files (the .py files) at the root level of this directory. 3. Install any libraries using pip at the root level of the directory. 4. Zip the content of the project-dir directory)

And I have also read this documentation: https://docs.aws.amazon.com/lambda/latest/dg/vpc-rds-deployment-pkg.html

The same applies to other modules or libraries e.g. openpyxl etc. I was always told that "No Module Named 'OneNameThatIHaveNotImported'"

So does anyone have any idea or who know another way how can one via lambda-function edit the csv-file on s3 and import the edited version into rds-database?

Thanks for the help in advance!

Upvotes: 0

Views: 2496

Answers (1)

Nich Chung
Nich Chung

Reputation: 1

the answer thread this SO answer references will put you on the right path. basically, you'd need to create the deployment package in an EC2 that matches the linux image the AWS lambda functions runs on. better yet, you can deploy lambda functions from the same staging EC2 instance where you created your deployment package through the AWS CLI.

you can also use [precompiled lambda packages][2] if you want an out-of-the-box fix.

[2]: https://github.com/jkehler/awslambda-psycopg2 or more generally, https://github.com/Miserlou/lambda-packages

Upvotes: 0

Related Questions