CODEWITHSUNDEEP
javaamazon-s3apache-camel
KingTravisG
KingTravisG

Reputation: 1336

Apache Camel - aggregate messages based on body

I'm currently using apache camel to consume SQS messages and everything has been working fine.

As part of one project, I consume S3 notification events when a file is uploaded. The files are uploaded to a predicable S3 key - (<type>/<account-id>/<id>/<file>).

When processing, I am using camel to aggregate messages into a single exchange (wait for either 10 messages, or timeout after 1 second). I was wondering but, is there a way to aggregate based on the S3 - for example, aggregate messages that have the same type, or id.

From what I understand from reading the camel documentation, there are ways to query Json payloads or header values - is this a possible approach (since the S3 event notification is a Json message, and according to the AWS documention a PUT operation would only generate a single record entry)? Or would I need to implement my own Aggregator?

To add a bit of context - I have a service which collects data, and uploads the data to S3. Another service will then download this data when notified, process it and uploads to another bucket. If I can aggregate S3 notifications, I am able to combine data and upload it, cutting down on the amount of uploads and API calls etc.

Upvotes: 0

Views: 1070

Answers (2)

KingTravisG
KingTravisG

Reputation: 1336

While not the best or most generic solution, I did find a way to get this working -

I simply added a additional Processor that gets invoked before passing to the aggregator. The processor simply checks the event record (since I am listening for PUT events from S3, according to the AWS docs there should only be one record) for the S3 key and sets the headers on the Message.

The Aggregator is then able to combine Exchanges based on these headers (simply S3-Type, S3-Account-Id, and S3-Id).

Upvotes: 0

mgyongyosi
mgyongyosi

Reputation: 2667

If you use the camel-aws s3 component then you can access/get the S3 key from the Message's CamelAwsS3Key header and you don't need to query the body, but you need to extract the required field from the S3 key.

Upvotes: 1

Related Questions