Reputation: 3186
I'm looking out for a solution to pipe DynamoDB Streams to my own webhook or on-prem data pipeline to process the data. Based on my exploration and also this thread, it looks like we need to connect DynamoDB streams to either
I can't figure out another adaptor or to be more specific a generic webhook adaptor.
Is it possible to pipe the stream outside by directly hooking to my webhook?
Upvotes: 1
Views: 1191
Reputation: 675
2 part series on stream processing from Dynamodb to Elasticsearch:
https://medium.com/realtime-data-streaming/data-streaming-from-dynamodb-scaling-it-up-8273d23295c
The first post has some evaluation on Lambda vs KCL and both has details on the KCL approach. Hope this helps.
Upvotes: 0
Reputation: 13055
Here is how I did it.
DynamoDB --> Streams (Old and New Image) --> Lambda --> (http/https or any other endpoint)
Have your DynamoDB Table create streams with old and new image of the changed data and send it to your lambda. Once the lambda is triggered it will deliver the packet for you.
You can also configure one or more batch data delivered to your lambda.
DynamoDB Streams:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html
NodeJS Lambda DynamoDB Events Trigger Implementation:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.Lambda.Tutorial.html
Hope it helps.
Upvotes: 1