CODEWITHSUNDEEP
apache-kafkamicroservices
Pavani
Pavani

Reputation: 31

data sync between 2 instances of same microservice using kafka

We have a microservice acts as a cache service and decided to have only 2 instances of this microservice up and running. This microservice receives data through kafka topic and stores in it as in memory cache. But we are having a challenge to sync data between these 2 microservices. We decided to use different consumer group for each instance to receive same data, so that, both instances will be in sync. Being same codebase, how to achieve subscribing to different consumer group during startup. For example, if instance#1 subscribes to consumergrp1, other instance2 should be able to subscribe to consumergrp2. Please suggest me how to achieve this.

Upvotes: 1

Views: 1171

Answers (2)

Vaibs
Vaibs

Reputation: 1606

You can not sync in-memory data in microservices for multiple instance when you are getting data from streaming system or it's getting multiple times.If you are getting data only once in pod life, then you can achieve the sync in-memory data. For e,g. while service is getting up, you can get the data from source and persist in-memory.In this case both pod is having the same data.

You need to use the distributed cache database like redis, couchbase cache.That will be the more clean and neat approach for this.

Upvotes: 2

Mark Bramnik
Mark Bramnik

Reputation: 42431

You haven't specified any details about the way you use kafka (language/thirdparties), etc. So, speaking "in general", you can:

  • specify a random (or partially random) consumer group id. It won't be as "clean" as "consumergrp1" and "conumergrp2", but its a string after all, so you can generate it randomly. This idea includes generating the identification of the process in a name of consumer group, for example, if the microservice instances are supposed to be running on different machines, you could include the name of machine as a part of the name of the consumer group.

  • More complicated, but still: if you have some shared storage, you could use it as a "synchronization" and store the monotonically increasing counter of the "current consumer group to create". once the value is read, it has to be increased. Of course the implementation details depend on the shared storage you actually use (DB, stuff like Redis, whatever).

So there are many different possible solutions. As a suggestion, in any solution you take, do not rely on the fact that you have exactly two instances of the service, maybe you'll reconsider that in future.

Upvotes: 1

Related Questions