CODEWITHSUNDEEP
javaapache-kafka
Tomasz S
Tomasz S

Reputation: 202

Kafka producer throws "Received unknown topic or partition error" when sending to topic created via AdminClient createTopics method

I have issue with topics created using AdminClient createTopics. In my application I have following sequence:

  1. create new topic with 1 partition, replication factor set to 1 using AdminClient.createTopics
  2. wait onAdminClient.createTopics KafkaFuture result
  3. immediately send a new message to newly created topic (usually the time between operation 2 and 3 is about 200 milliseconds).

My code is as follows:

adminClient
  .createTopics(Collections.singleton(new NewTopic(targetTopic, 1, (short) 1)))
  .values()
  .get(targetTopic)
  .get();
producer.send(new ProducerRecord<>(targetTopic, data));

From time to time producer doesn't see the created topic and throws following exception:

[Producer clientId=producer-1] Error while fetching metadata with correlation id 5 : {targetTopic=UNKNOWN_TOPIC_OR_PARTITION}

[Producer clientId=producer-1] Received unknown topic or partition error in produce request on partition targetTopic. The topic/partition may not exist or the user may not have Describe access to it

This issue is very rare (< 0,1% all created topics).

Is it guaranteed that when AdminClient.createTopics Kafka future is completed then topic is created and Kafka producer should see that topic ? If not then which method of topic creation can give me such guarantee ?

I am using kafka-clients:2.0.0 and Kafka HD service on Azure. My cluster consists of 3 Zookeeper and 3 Kafka nodes.

Upvotes: 7

Views: 29822

Answers (2)

Roshith S
Roshith S

Reputation: 351

Alert - This might not be the solution for all since there are many possibilities for an "Unknown topic or partition error". Try to confirm the below configurations :

  • The topic name spelled correctly and the topic exists.
  • The consume() method can work with multiple topics. So if you have assigned the list of topics as comma separate values in a single string - make sure you convert it to List before passing to consume. This solved my issue.

Upvotes: 0

Burak Akyıldız
Burak Akyıldız

Reputation: 1644

Kafka client is not guarenteed about atomic topic creation. You can read it from this documentation. When you create a topic, kafka will notify distributed systems about topics information so the action is running on different machines and these can't be atomic without distributed transactional management.

Upvotes: 3

Related Questions