CODEWITHSUNDEEP
restapache-kafkaspring-kafkamessage-forwarding
Eric
Eric

Reputation: 469

Forwarding messages coming from Kafka Topic

I am in the process of designing a system which acts like a message forwarder from one system to another system. I have several options to go for but I would like to apply the best option which provides less resource consumption (cpu, ram) and latency. Thus, I need your recommendation and view on this. We assume that messages will be streaming to our system from a topic in Kafka. We need to forward all the messages from the topic to another host. There can be different strategies for this purpose.

  1. Collect certain number of messages let's say 100 messages (batch processing) and send them at once within a single HTTP message.
  2. When one message is received, system will send this message as the http POST request to the target host.
  3. Open webSocket between our system and the target host and send messages.
  4. Behave like a Kafka producer and send messages to topic.

Each of them might have advantages and disadvantages. I have concern that system may not handle the high amount of messages coming. Do you have any option other than these 4 items? Which is the best option you think in terms of what?

Upvotes: 0

Views: 1545

Answers (2)

Raul Lapeira Herrero
Raul Lapeira Herrero

Reputation: 987

I think you are looking for Kafka Streaming in this scenario. Although from an efficiency point of view maybe some hadoop stack implementation (Flume) or Spark would be less consuming, not sure, depends on amount of data, network jumps, disk used, amount of memory.

If you have huge amounts of messages those distributed solutions should be your right approach not a custom REST client.

Upvotes: 0

Yannick
Yannick

Reputation: 1418

How important your latency requirement is ? HTTP is quite slow, compared to an UDP based messaging system, but maybe you don't need a so tailored latency. Batching your messages will increase latency, as you may know.

But it's disturbing because the title of this page is "rest - forwarding" =). Does it has to be REST ( so HTTP) ? because it seems you can as well envisage to act like a kafka producer, if so, it's not REST.

The memory footprint of Kafka may be a bit high (Java lib), but no so much. Do you work on embedded system (willing to reduce memory footprint ?)

For CPU purposes.. it depends with what we're comparing Kafka, but I still think Kafka is quite optimised when asking for performance.

Think we lack more information about this "another host" , could you give more details about its purpose ?

Yannick

Upvotes: 0

Related Questions