CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
Martin Kamp Jensen
Martin Kamp Jensen

Reputation: 157

Kafka Streams 2.5.0 requires input topic

Starting with Kafka Streams 2.5.0 it seems like a topology must include an input topic. In Kafka 2.4.1 (and earlier) that is not the case.

I have an application where the topology is just creating a few global state stores that read in data from topics written to by other applications.

With Kafka 2.5.0 I get this error:

13:24:27.161 [<redacted>-7cf1b5c9-4a6e-4bf2-9f77-f7f85f2df3bb-StreamThread-1] ERROR o.a.k.s.p.internals.StreamThread - stream-thread [<redacted>-7cf1b5c9-4a6e-4bf2-9f77-f7f85f2df3bb-StreamThread-1] Encountered the following error during processing:
java.lang.IllegalStateException: Consumer is not subscribed to any topics or assigned any partitions
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1228)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:853)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:753)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:697)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670)

If I add a dummy input topic (e.g. via streamsBuilder.stream(Pattern.compile("hack"));) the application starts fine.

Is this behavior to be expected or is it an unintentional change in Kafka Streams 2.5.0?

More details: The use case above may seem a bit weird and I would have to agree. The reason for doing it in the first place was a shortcoming of Interactive Queries where for periods of time the application could not answer queries. I see that issue has been fixed in Kafka Streans 2.5.0 via KIP-535 which is great. I hope to look into IQ again later.

Upvotes: 6

Views: 4434

Answers (3)

S Blee-G
S Blee-G

Reputation: 166

There was a regression introduced in 2.5.0 when we switched (back) to using collection subscription. A fix was just merged, so you should upgrade to 2.5.1 or 2.6 when they are released.

Upvotes: 3

S Blee-G
S Blee-G

Reputation: 166

If you don't have any non-global parts of your topology, there's no reason to have any StreamThreads at all. Meaning you can easily work around this by just setting num.threads to zero -- arguably, you should be doing this anyways to avoid unnecessary overhead and group coordination. Setting this to zero by default when a global-only topology is detected is the "fix" anyway, so you don't need to wait for that

Upvotes: 3

pjjohnston
pjjohnston

Reputation: 1

private void subscribeConsumer() {
    if (builder.usesPatternSubscription()) {
        // this is old behaviour - is there a config that will revert to this??
        consumer.subscribe(builder.sourceTopicPattern(), rebalanceListener);
    } else {
        consumer.subscribe(builder.sourceTopicCollection(), rebalanceListener);
    }
}

Upvotes: 0

Related Questions