How to handle AWS IOT streaming data in relational database

Generic information :-i am designing solution for one of IOT problem approach in which data is continuously streaming from plc(programmable logic controller),plc have different tags these tags are representation of telemetry data and data will be continuously streaming from these tags, each of devices will have alarm tags which will be 0 or 1 , 1 means there is an equipment failure problem statement:- i have to read the alarm tag and raise a ticket if any of alarm tag value is 1 and i have to stream these alerts to dashboard and also i have to maintain the ticket history too,so the operator can update the ticket status too

My solution:- i am using aws IOT , i am getting data in dynamo db then i am using dynamo db stream to check if any new item is added in alarm table and if it will trigger lambda function (which i have implemented in java) lambda function opens a new ticket in relational database using hibernate.

problem with my approach:-the aws iot data is continuously streaming in alarm table at a very fast rate and this is opening a lot of connection before it can be closed that's taking my relational database down

please let me know if other good design approach can i adopt?

Upvotes: 2

Answers (4)

winter

Reputation: 144

You can control number of lambda function concurrency. And this will reduce the number of lambdas that get spinned up based on the dynamo events. Thereby reducing the connections to RDS.

https://aws.amazon.com/blogs/compute/managing-aws-lambda-function-concurrency/

Ofcourse , this will throttle the dynamo events.

Upvotes: 0

Aniket Sinha

Reputation: 6031

I agree with raevilman's design of not letting Lambda contact RDS directly. Since creating a new ticket is not the only task you Lambda function is doing, you are also streaming these alerts to a dashboard. Depending on the streaming rate and the RDS limitations, you may want to split these tasks in multiple queues.

Generic solution: I'd suggest you can push the alarm to a fanout exchange and this exchange will in turn push the alarm to one or more queues as required. You can then batch the alarms and perform multiple writes together without performing connect/disconnect cycle multiple times.

AWS specific Solution: I haven't used SQS so can't really comment on it's architecture. Alternatively, you can create an SNS Topic and publish these alarms to this topic. You can then have SQS queues as subscribers to this topic which in turn will be used for Ticketing and Dashboard purpose independent of each other.

Here again, from Ticketing queue, you can poll messages using Lambda or your own scheduler in batch and process tickets(frequency depending on how time critical alarms are). You may want to read this tutorial to get some pointers.

Upvotes: 1

raevilman

Reputation: 3259

Just a proposal....

From lambda, do not contact RDS,

Rather push all alarms in AWS SQS

then you can have one another lambda scheduled for every minute using AWS CloudWatch Rules that will pick all items from AWS SQS and then insert them in RDS at once.

Upvotes: 1

raevilman

Reputation: 3259

USE Amazon Kinesis Analytics to process streaming data. Dynamodb isn't suitable for this.

How to handle AWS IOT streaming data in relational database

Answers (4)

Related Questions