CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
Saloni Vithalani
Saloni Vithalani

Reputation: 323

How many kafka streams app is recommended to run on single machine in production?

In our architecture, we are assuming to run three jvm processes on one machine (approx.) and each jvm machine can host upto 15 kafka-stream apps.

And if I am not wrong each kafka-stream app spawns one java thread. So, this seems like an awkward architecture to have with around 45 kafka-stream apps running on a single machine.

So, I have question in three parts

1) Is my understanding correct that each kafka-stream app spawns one java thread ? Also, each kafka-stream starts a new tcp connection with kafka-broker ?

2) Is there a way to share one tcp connection for multiple kafka-streams ?

3) Is is difficult(not recommended) to run 45 streams on single machine ? The answer to this is definitely NO unless there is a real use case in production.

Upvotes: 0

Views: 1543

Answers (2)

Kiran Balakrishnan
Kiran Balakrishnan

Reputation: 257

  • Each Kafka stream job spawns a single thread.If the thread number is set as n numbers it will provide parallelism in processing n number
    of Kafka partitions.
  • If a single machine does not have the capacity to run large number of threads, parallelism can be achieved by submitting the Streams
    applications job with same application name in another machine in the same cluster. The job will be identified by Kafka
    streams and handled in background.
  • is is difficult(not recommended) to run 45 streams on single machine ? The answer to this is definitely NO unless there is a real use case in production.--unless your system has these many cores or the input has 45 partition this is not necessary

Upvotes: 0

Matthias J. Sax
Matthias J. Sax

Reputation: 62350

Multiple answers:

  • a KafkaStreams instance start one processing thread by default (you can configure more processing threads, too)

  • internally, KafkaStreams uses two KafkaConsumers and one KafkaProducer (if you turn on EOS, it uses even more KafkaProducers): a KafkaConsumer starts a background heartbeat thread and a KafkaProducer starts a background sender thread => you get 4 threads in total (processing, 2x heartbeat, sender) -- if you configure two processing threads, you end up with 8 threads in total, etc)

  • there is more than one TCP connection as the consumer and the producer (and the restore consumer, if you enable StandbyTasks) connect to the cluster

  • it's not possible to share any TPC connections atm (this would require a mayor rewrite of consumers and producers)

  • how many threads you can efficient run depends on your hardware and workload... monitor you CPU utilization and see how buys your machine is...

Upvotes: 3

Related Questions