CODEWITHSUNDEEP
goapache-kafka-streams
tmp dev
tmp dev

Reputation: 9219

kafka streams in golang

I'm trying to use golang to create a kafka stream client in Go. From what I have seen this is only possible if using a Java Client. I did a bit of searching and found a few other third party libraries but nothing official. Also from my limited understanding I think streams is syntactic sugar over the standard consumers ? is this correct ?

Upvotes: 3

Views: 9931

Answers (2)

runwuf
runwuf

Reputation: 1717

When using standard Consumer and Producer directly, it lacks support for such as the following:

  • stream transformation (aggregation and etc.)
  • fault-tolerant state in local (kstreams uses RocksDB)

Goka [https://github.com/lovoo/goka] is now a relatively active project for Kafka Streams golang support.

Upvotes: 1

Muhammad Waqas Dilawar
Muhammad Waqas Dilawar

Reputation: 2342

To answer your this particular question,

Also from my limited understanding I think streams is syntactic sugar over the standard consumers ? is this correct?

When implementing asynchronous microservices, we could use producer and consumer APIs but these APIs are too low level, they were good to understand how to use Kafka, but if we want to implement more complex applications, they might be too low level. Also, when we develop event‑driven applications, we might need to implement multistage processing when we would connect multiple stages. At each stage, we will read events from Kafka, process them, and write to output topics. Again, using producer and consumer APIs would be quite a lot of work. And one of the more high‑level solutions we can use is called Kafka Streams. Kafka Streams is a very versatile library, it supports stateless stream processing and it also supports stateful processing.

Note: And if you have the option of working with a language other than Go, I would highly recommend working with Java for Kafka Streams. Just to mention here we have been working with Kafka Streams using Java for the last two years and we have felt Kafka Streams is more of a Java library than a distributed system.

Upvotes: 0

Related Questions