CODEWITHSUNDEEP
eventsstreamtime-seriesmqttiot
jianrui
jianrui

Reputation: 151

Best practice for IoT stream data processing

I assume that there are hundreds and thousands IoT devices that publish the data to the (broker)MQTT cluster via the MQTT protocol, behind the broker i have the data processing module which subscribe the data from the broker and maintain a status table for all these devices. The number of the devices is still rising, therefor I have to scale out the broker cluster and data processing module accordingly, for the MQTT broker such as Kafka/Rabbit MQ/Hive MQ can be scaled out very easily, but for the data processing module I'm not quite sure whether there is any best practice, or any framework/architecture can achieve this very easily:

I assume I have to create many daemon processes with hundreds and thousands threads to listen on the MQTT broker, the question is how to scale out these services dynamically?

Thanks.

Upvotes: 3

Views: 879

Answers (1)

Mehdi
Mehdi

Reputation: 196

One way of doing this would be using Node.js as it uses an event-driven approach and you don't have to deal with threads, etc.

I found this library for Node.js which is specific to MQTT: https://www.npmjs.com/package/mqtt You can use this to subscribe to different topics.

You may also find this project interesting: http://nodered.org/

The other solution can be using Apache Kafka which has scalability as an important feature. However, the problem here is that Kafka does not support MQTT out of the box and has its own conventions. Therefore, there is a need for some sort of adapter to make them work together. For that, take a look at this: using mqtt protocol with kafka as a message broker

Upvotes: 1

Related Questions