CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3aws-lambdaboto
Data_101
Data_101

Reputation: 953

AWS Lambda in Python to copy new files to another s3 bucket

I have created a Lambda which is triggered every time I add a file to s3://test-bucket-01/, it copies all the files in s3://test-bucket-01/ to s3://test-bucket-02/

I would like it to only add the new file that has just been added?

Current Code:

import boto3
s3 = boto3.resource('s3')


def lambda_handler(event, context):
    bucket = s3.Bucket('test-bucket-01')
    dest_bucket = s3.Bucket('tb-bucket-02')
    print(bucket)
    print(dest_bucket)

    for obj in bucket.objects.all():
        dest_key = obj.key
        print(dest_key)
        s3.Object(dest_bucket.name, dest_key).copy_from(CopySource = {'Bucket': obj.bucket_name, 'Key': obj.key})

Results:

Function Logs:
START RequestId: XXXXXXX-XXXXXXX-XXXXXXX Version: $LATEST
s3.Bucket(name='test-bucket-01')
s3.Bucket(name='test-bucket-02')
test-data-01.json
test-data-02.json
test-data-03.json

Upvotes: 0

Views: 10724

Answers (3)

Anish Gopinath
Anish Gopinath

Reputation: 182

so what is the answer ? how to trigger the lambda only when a new file is added?

Upvotes: 0

anton.uspishnyi
anton.uspishnyi

Reputation: 352

Why don't you want to use native functional of AWS S3 to replicate new files from one bucket to another?

It's easier and more reliable. And you can replicate files by prefixes, change storage plans and many others with this native feature.

Upvotes: 0

romruben
romruben

Reputation: 194

You not need to list the bucket to copy all files because the event variable is a json that contains the file that you has been copied into test-bucket-01.

Example:

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "EXAMPLE"
      },
      "requestParameters": {
        "sourceIPAddress": "127.0.0.1"
      },
      "responseElements": {
        "x-amz-request-id": "C3D13FE58DE4C810",
        "x-amz-id-2": "FMyUVURIY8/IgAtTv8xRjskZQpcIZ9KG4V5Wp6S7S/JRWeUWerMUE5JgHvANOjpD"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "testConfigRule",
        "bucket": {
          "name": "sourcebucket",
          "ownerIdentity": {
            "principalId": "EXAMPLE"
          },
          "arn": "arn:aws:s3:::mybucket"
        },
        "object": {
          "key": "key/to/HappyFace.jpg",
          "size": 1024,
          "eTag": "d41d8cd98f00b204e9800998ecf8427e"
        }
      }
    }
  ]
}

In this case, this event notify ObjectCreated:Put event. A new file has been copied to "sourcebucket" bucket with key "key/to/HappyFace.jpg".

Upvotes: 2

Related Questions