CODEWITHSUNDEEP
c#apache-kafkaconfluent-schema-registry
user6424058
user6424058

Reputation: 3

Kafka Producer Error --> System.InvalidOperationException

So I am sending about 300 records every 3 seconds and I am seeing error message : System.InvalidOperation:Failed to create thread: Result too large(#) at Confluent.Kafka.ProducerBuilder`2.Build()

I Believe this is caused because there are too many messages in the Queue , is there a way to resolve this issue?

Code Snippet:

await producerBuilder(topic,new Message<Null,object>)
.ContinueWith(
task => {
if(task.IsFaulted) 
  {
   //Do Something
  }
}
producerBuilder.Flush(FlushTimeout);

Upvotes: 0

Views: 50

Answers (1)

Ivan Petrov
Ivan Petrov

Reputation: 4350

Each Producer has as an implementation detail - a dedicated thread to itself for polling, which is created when it is constructed -> prodBuilder.Build()

=> Task.Factory.StartNew(() => {
    try {
        ....
}, ct, TaskCreationOptions.LongRunning, TaskScheduler.Default);

which could explain part of the exception message you are getting:

System.InvalidOperation:Failed to create thread:

So my suspicion is that you are creating fresh producers which you do not dispose properly, so available threads for polling are exhausted.

There is also this github issue for the librdkafka library that Confluent.Kafka is a wrapper around - where it's hinted that too many producers may produce similar errors:

Any way, librdkafka creates a number of threads per instance internally and with your number of instances you are hitting the thread / proc limit. See here: Maximum number of threads per process in Linux?

Upvotes: 1

Related Questions