CODEWITHSUNDEEP
goapache-kafkaconfluent-kafka-go
yuyang
yuyang

Reputation: 1671

How can we ensure reliable and low-latency kafka writing using confluent-kafka-go client?

The sample producer code at https://github.com/confluentinc/confluent-kafka-go calls Producer.Produce(...) It writes the message to the sending buffer queue and return immediately. However, writing to the internal buffer queue does not guarantee successful message delivery. How can we ensure sucessful message delivery here?

for idx, word := range []string{"Welcome", "to", "the", 
                                "Confluent", "Kafka", "Golang", "client"} {
    start_time := time.Now().UnixNano()
    p.Produce(&kafka.Message{
        TopicPartition: kafka.TopicPartition{Topic: &topic, 
                                             Partition: kafka.PartitionAny},
        Value:          []byte(word),
    }, nil)
}

Upvotes: 1

Views: 583

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191973

To block and send data immediately, you'd call Flush method of the producer

https://github.com/confluentinc/confluent-kafka-go/blob/master/kafka/producer.go#L335

To ensure it's written reliably, set acks=all producer config and ensure the topic is configured to have min in sync replicas as 2 or more, and topic replication factor as 3 or more

Upvotes: 1

Related Questions