If I use Kafka as simple message. Does it really worth

Question

=== Assume everything from consumer point of view ===

1. I was reading couple of Kafka articles and I saw that the number of partitions is coupled to number of micro-service instances.... Ex: If I say 1topic 1partition for my serviceA.. Producer pushes message to topicT1, partitionP1, and from consumerSide(ServiceA1) I can read from t1,p1. If I spin new pod(ServiceA2) to have highThroughput then second instance will never receive any message because Kafka/ZooKeeper assigns id to each Consumer and partition1 is already taken by serviceA1. So serviceA2++ stays idle... To avoid such a hassle Kafka recommends to add more partition, so that number of consumers can be increased/decreased based on need.
2. I was also able to test through commandLine and service2 never consumed any message. If I shut service1 then service2 was able to pick new message... So if I spin more pod then FailSafe/Availability increases but throughput is same always...
3. Is my assumption is correct. Am I missing anything. Now I feel like any standard messaging will have the same problem...How to extend message-oriented systems itself.

Answer 1

1. We just started replacing messaging with Kafka.

2. In a traditional MQ there will be a cluster and 1orMQ will be there inside.

3. So the MQ cluster/co-ordinator service will deliver the message to clients.

4. Now there can be 10 services/clients which can consume message from single MQ.

5. So if there are 10 messages in MQ then each service/consumer/client can read/process 1 message

6. Now this case is not possible in Kafka which I understood now as per design

7. To achieve similar functionality in Kafka I have add equal or more number of partition as client/consumer/pods.

Answer 2

A broad solution to this is to decouple consumption of a message (i.e. receiving a message from Kafka and perhaps deserializing it and validating that it conforms to the schema) and processing it (interpreting the message). If the consumption is simple enough, being limited to no more instances consuming than there are partitions need not constrain.

One way to accomplish this is to have a Kafka consumption service which sends an HTTP request (perhaps through a load balancer or whatever) to a processing service which has arbitrarily many members.

Note that depending on what you're using Kafka for, there may be a requirement that certain messages always be in the same partition as one another in order to ensure that they get handled in a deterministic order (since ordering across partitions is not guaranteed). A typical example of this would be if the messages are change events for a particular record. If you're accomplishing this via some hash of the message key (or a portion of the key if using a custom partitioner), then simply changing the number of partitions might not be viable (you would need to introduce some sort of migration or have the producers know which records have to be routed to the old partitions and only route to the new partitions if the record has never been seen before).

Answer 3

Every topic has a partition, by default it comes with only one partition if you don't define the partition count value. In your case, you have a consumer group that consists of two consumers. Every consumer read the log from the partition. In your case, first consumer read the log from the first partition(we have the only partition), and for second consumer there will be no partition to the consumer the data so it become idle. Once first consumer gets down then only the second consumer starts reading the data from the first partition from the last committed offset.

Please check below blogs and videos. It explains the topic, consumer, and consumer group in kafka.

https://www.javatpoint.com/apache-kafka-consumer-and-consumer-groups http://cloudurable.com/blog/kafka-architecture-consumers/index.html https://docs.confluent.io/platform/current/clients/consumer.html https://www.youtube.com/watch?v=lAdG16KaHLs

I hope this will give you idea about the consumer and consumer group.