CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-sagemakeraws-event-bridge
user1114
user1114

Reputation: 1169

AWS EventBridge: Add multiple event details to a target parameter

I created an EventBridge rule that triggers a Sagemaker Pipeline when someone uploads a new file to an S3 bucket. As new input files become available, they will be uploaded to the bucket for processing. I'd like the pipeline to process only the uploaded file, and so thought to pass in the S3 URL of the file as a parameter to the Pipeline. Since the full URL doesn't exist as a single field value in the S3 event, I was wondering if there is some way to concatenate multiple field values into a single parameter value that EventBridge will pass on to the target.

For example, I know the name of the uploaded file can be sent from EventBridge using $.detail.object.key and the bucket name can be sent using $.detail.bucket.name, so I'm wondering if I can send both somehow to get something like this to the Sagemaker Pipeline s3://my-bucket/path/to/file.csv

For what it's worth, I tried splitting the parameter into two (one being s3://bucket-name/ and the other being default_file.csv) when defining the pipeline, but got an error saying Pipeline variables do not support concatenation when combining the two into one.

The relevant pipeline step is

step_transform = TransformStep(name = "Name", transformer=transformer,inputs=TransformInput(data=variable_of_s3_path)

Upvotes: 1

Views: 3771

Answers (1)

fedonev
fedonev

Reputation: 25649

Input transformers manipulate the event payload that EventBridge sends to the target. Transforms consist of (1) an "input path" that maps substitution variable names to JSON-paths in the event and (2) a "template" that references the substitution variables.

Input path:

{
  "detail-bucket-name": "$.detail.bucket.name",
  "detail-object-key": "$.detail.object.key"
}

Input template that concatenates the s3 url and outputs it along with the original event payload:

{
  "s3Url": "s3://<detail-bucket-name>/<detail-object-key>",
  "original": "$"
}

Define the transform in the EventBridge console by editing the rule: Rule > Select Targets > Additional Settings.

Upvotes: 3

Related Questions