CODEWITHSUNDEEP
amazon-s3amazon-sqs
barath
barath

Reputation: 842

S3 event notification only sends metadata to SQS

Trying to understanding about SQS.

I recently enabled an S3 event to sync data form S3 to SQS. When I upload a file (some plain text data) to S3 I only see metadata (of the file uploaded to S3) in the SQS queue and not the actual data in the file.

Below is response I get from SQS cli.

{
"Messages": [{
    "Body": "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-west-2\",\"eventTime\":\"2019-02-03T04:20:52.812Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"AWS:AROAJPPH7T6UMTQNWQDV4:[email protected]\"},\"requestParameters\":{\"sourceIPAddress\":\"***.***.95.237\"},\"responseElements\":{\"x-amz-request-id\":\"D0FBEB5DD148EC84\",\"x-amz-id-2\":\"F7q+hW5hb6f02J42gT1wWDWjAp2wuQ3ojio1n2Bz/jZCS+e+6/zjkYsUfRNB59dqH61p67rPJDU=\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"SQS-hops-mab-new-hotelids\",\"bucket\":{\"name\":\"bukcet-name\",\"ownerIdentity\":{\"principalId\":\"A3HAX2MIKOSGAO\"},\"arn\":\"arn:aws:s3:::bukcet-name\"},\"object\":{\"key\":\"prefix/data_file\",\"size\":80,\"eTag\":\"b10063d01db20998584a425f0fa77a29\",\"sequencer\":\"005C566C24C2E9A48A\"}}}]}",
    "ReceiptHandle": "AQEBGIumiawoH/pKK5deThbTsCuZ8bWlgF/p8qiL1VTrDqpkUkVxWUxlzMftEtfRzqFZKx+nyUCjuSb50Oicd7hq51LUexOro0+GTVtQsxJlEJ9RaxIk+7/Ia1Ac7ox2G4Mt1nnFBDiEsiPgwYwbQe0fAz1EFiI3QZuZtOdyPNFAIaObPJGhF591IwSA6xhIqgHYJwo5pvw/axVCd05Ex34wg/TFIN9Ha4RVUSDWFXpBmPtM/1V6Ek6HP/Ag0Ps3HKhG1KNOFUjq0mv6bGqaXI/HuxCGitj3uk0xX25y0lT+RFomWcDyKpbmfm2fue8StABrKE7sm/inc1Zbd/4qKFTNUdPqQVqIr3s55plAPDXGvNiXBx4O7FkrE1jrxHBeONMRrlYHe2/KeL0WaITYu3dxNw==",
    "MD5OfBody": "0c04939f7c38ff3f9992ef911b514ed7",
    "MessageId": "f7fcb7da-66cb-4a59-acf0-79f75e9a60ce"
}]

}

Below is the metadata in the above Body section.

{
"Records": [{
    "eventVersion": "2.1",
    "eventSource": "aws:s3",
    "awsRegion": "us-west-2",
    "eventTime": "2019-02-04T01:00:31.166Z",
    "eventName": "ObjectCreated:Put",
    "userIdentity": {
        "principalId": "AWS:AROAJPPH7T6UMTQNWQDV4:email@****.com"
    },
    "requestParameters": {
        "sourceIPAddress": "***.***.95.237"
    },
    "responseElements": {
        "x-amz-request-id": "24703DDF194A1F7E",
        "x-amz-id-2": "wwpJeU6B3yZT7qaB7imsucsaM8y1hpFBi14POL4L1/2C1/CoBRkdHLgIOPzyy85a3RPJQFfE9Ys="
    },
    "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "SQS-name",
        "bucket": {
            "name": "bucket-name",
            "ownerIdentity": {
                "principalId": "A3HAX2MIKOSGAO"
            },
            "arn": "arn:aws:s3:::bucket-name"
        },
        "object": {
            "key": "prefix/datafile",
            "size": 80,
            "eTag": "b10063d01db20998584a425f0fa77a29",
            "sequencer": "005C578EAF156E8A91"
        }
    }
}]

}

Doesn't the S3 event notification send the actual data to SQS? Am I missing something?

Upvotes: 1

Views: 2660

Answers (1)

A.Khan
A.Khan

Reputation: 3992

Yes this is correct SQS will only get the metadata of S3 object being created/modified or based on notification event. Regardless of the destination type you choose, S3 notification structure is same for all destinations. S3 object could be anything audio, video or some huge JSON file. SQS has its limits and it cannot always receive actual S3 object.

Upvotes: 2

Related Questions