CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
gwk
gwk

Reputation: 268

How does the Kafka Streams DSL process method work?

I've been working with Kafka Streams a bit and got some basic functionality working but I'm having some trouble wrapping my head around the process method in the Kafka Streams DSL. Specifically:

I know there are two ways of using Kafka Streams, the lower-level Processing API and the higher level Streams DSL. With the lower level you more explicitly define your topology, naming each node and such while the Streams DSL abstracts most of that away.

However, the higher level Streams DSL has a method called process() which is a terminal operation (i.e. the method returns void). So my question is, where does the processed data - the data that the Processor's void forward(key, value) method sends - go?

In the lower level Processing API, you name you processor Node and can link Sinks to that name but in the Streams DSL there is no name, or at least none that I can find.

Upvotes: 0

Views: 624

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62285

It is possible but "clumsy" to forward() data from a Processor in the DSL. As you stated, the process() method is defined as terminal operation and thus, you should not call forward(). If you call forward() anyway and there is no downstream processor assigned, forward() is basically a no-op.

However, instead of trying add a downstream processor for process() (what would be a hack), you should use transform() (and related methods, transformValues(), flatTransform(), and flatTransformValues()) instead ff you want to send data down stream.

Upvotes: 1

Related Questions