Does RabbitMQ or Kafka fit for real time dashboard applications?

Question

We are facing a critical decision in which we need to choose the right tool for a real time telemetry processing and monitoring system.

The system has two main purposes:

1. Displaying real time telemetry data in a dashboard UI application (high refresh rate)
2. Process large amount of data from multiple sources

We have two concerns:

• In our use-case, most of the data consumers will be web clients in contrast to what we discovered so far -> both RabbitMQ and Kafka are mostly used in cloud based platforms in which most of the data consumers are services and not clients.

• Usually data consumers from Kafka or RabbitMQ are load balanced and messaged are not replicated for each client.

.

1. Does Kafka or RabbitMQ fit for environments where data is replicated for each client?
2. Does this kind of tools fit for handling large amount of clients while replicating messages for each client?
3. Is there a better tool for this kind of job?

Thank you :)

Answer 1

Honestly for monitoring I would also consider dedicated solutions like splunk or ELK Stack (Elasticsearch and family).

But IMHO Kafka will do the job in soft real-time mode (there are some latencies due to the polling).

Answers: 1. Yes. 2. Yes. 3. It depends. I was using as well p2p messaging like YAMI4. This one does not use polling so it is great, when low latencies are needed, but YAMI4 has no implemented persistency.

Answer 2

Something to also consider in your technology choice is that Kafka also includes the Kafka Streams API, which lets you do stream processing within your application. It also includes a queryable statestore, which can be used to drive your dashboards.

Consumers in Kafka can opt to receive all messages, with a variety of native client libraries (Java, C/C++, python, Go, etc), and in addition a REST proxy for sending/receiving data from Kafka.

Answer 3

1. Kafka has partitioned topics, so each consumer group consumes messages independently. RabbitMQ has an exchange type called fanout that is used to deliver messages to all queues on the exchange, so that could be used for a similar purporse.

2. If you have a large number of clients (hundreds or thousands), I would not use a RabbitMQ fanout exchange. Kafka can scale horizontally and could handle a fairly large number of partitions. For latency purposes, it is generally recommended that you follow the rule of no more than 100 x b x r, where b is the number of brokers in a Kafka cluster and r is the replication factor.

3. There are other systems designed specifically for high-volume event processing. Have you looked at Event Store, or for cloud-based, Azure Event Hub?