CODEWITHSUNDEEP
apache-kafkaconfluent-platformkafka-rest
jlunavtgrad
jlunavtgrad

Reputation: 1015

How to handle back pressure with Kafka REST Proxy

I am creating a service that sends lots of data to kafka-rest-proxy. I am only sending data (producing) to kafka. What I'm finding is that kafka-rest-proxy is easily overwhelmed and runs out of java heap space. I've allocated additional resources, and even horizontally scaled out the number of hosts running kafka-rest-proxy, yet I still encounter dropped connections and memory issues.

I'm not familiar with the internals of kafka-rest-proxy, but my hunch is that it's buffering the records and sending them to Kafka asynchronously. If that is the case then what mechanism does it have to control back pressure? Is there a way to configure it such that it writes records to Kafka synchronously?

Upvotes: 1

Views: 739

Answers (1)

Rohit Yadav
Rohit Yadav

Reputation: 2578

Kafka REST Proxy exposes all of the functionality of the Java producers, consumer's command-line tools. Rest Proxy doesn't need any back pressure concept.

To be more specific, Kafka is capable of delivering messages over the network at an alarmingly fast rate.

You need to scale the brokers as per the rate you are producing and consuming the data.

Upvotes: 0

Related Questions