CODEWITHSUNDEEP
apache-kafkamessagingspring-kafka
Nafiul Alam Fuji
Nafiul Alam Fuji

Reputation: 434

One to One and Group Messaging using Kafka

As Kafka has a topic based pub-sub architecture how can I handle One-to-One and Group Messaging part of web application using Kafka? I am using SpringBoot+Angular stack and Docker Kafka server.

Upvotes: 6

Views: 6485

Answers (3)

PatrickChen
PatrickChen

Reputation: 1420

I'll write another answer here. Based on my experience with the chatting service. You only need one topic for all the messages. Using a well designed Message body. 

public class Message {
 private String from; // user id
 private String to;  // user id or group id
}

Then you can create like 100 partitions for this topic and create two consumers to consume them (50 partitions for one consumer in the beginning). Then if your system reaches the bottleneck, you can easier scale X more consumers to handle the load.

How to do distribute the messages in the consumer. I used to send the messages to the Mobile app, so all the app has a long-existing connection to the server, and the server sends the messages to the app by that channel. For group chat, I create a Redis cache to store all the active users in the group, so I can easier get the users who belong to this group, send them the messages.

And another thing, Kafka stateless, means Kafka doesn't de-coupled from the business logic, only acts as a message system, transfers the messages. If you connect your business logic to Kafka, like create a topic "One-to-One" and delete some after they finished, Kafka will be very messy.

Upvotes: 8

PatrickChen
PatrickChen

Reputation: 1420

Ok, It's a very complicated question, I try to type some simple basic information.

Kafka topics are divided into a number of partitions. Partitions allow you to parallelize a topic by splitting the data in a particular topic across multiple brokers — each partition can be placed on a separate machine to allow for multiple consumers to read from a topic in parallel.

So if you are using partitions, means you have multiple consumers to consume some in parallel.

consumer groups for a given topic — each consumer within the group reads from a unique partition and the group as a whole consumes all messages from the entire topic.

Basically, you can have only one group, then the message will not be processed twice in the same consumer group, and this is how Kafka delivers exactly once.

If you need two consumer groups, you need to think about why you need two? Are the consumers in two groups handling the different logic?

There is more, please check the official document, or you can answer a smaller question.

Upvotes: 0

JavaTechnical
JavaTechnical

Reputation: 9347

By One-to-One, I suppose you mean one producer and one consumer i.e. using at as a queue.

This is certainly possible with Kafka. You can have one consumer subscribe to a topic and and restrict others by not giving them authorization . See Authorization in Kafka

Note that once a message is consumed, it is not deleted, rather it is committed so that the same consumer will not consume it again.

By Group Messaging, I suppose you mean one producer > multiple consumers or multiple-producer > multiple-consumers

This is also possible, a producer can produce messages to a topic and multiple consumers can consume them.

If all the consumers have the same group id, then each consumer in the group gets only a subset of messages.

If they have different group ids then each consumer will get all messages.

Multiple producers also can produce to the same topic.

A consumer can also subscribe to multiple topics.

Upvotes: 2

Related Questions