Confluent's Kafka REST Proxy vs Kafka Client

Question

I am curious about the advantages and disadvantages of Confluent's Kafka REST Proxy and the producer/consumer implemented with the kafka official client library. i know that Confluent's Kafka REST Proxy is used for administrative tasks and for languages ​​not supported by the kafka client.

So, what are the advantages of the kafka client?

Answer 1

There are some architectures that don't lend themselves well to the asynchronous nature of the Kafka protocol. If the client is a web application, it won't have the kind of transport layer control required to connect directly to the Kafka cluster. If the client is built on Function as a Service technologies like AWS Lambda or Azure Functions, the overhead of creating connections to the cluster on startup, and the requirement to wait for outgoing messages to get accepted by the cluster, would be very inefficient and costly. In these cases, some form of proxy is really a necessity.

The Kafka client and protocol, when used correctly, provide some excellent features in terms of resiliency (retrying in the background to ensure messages are delivered despite short-term connection issues; communicating with multiple nodes and ensuring each message is accepted by enough of them) and performance (queueing and batching messages; using TCP directly; more efficient data serialization; skipping the man-in-the-middle), so it's generally a good idea to use the Kafka client directly if you're in a server-like environment that has a long lifecycle and technologies with good Kafka client implementations available to you.

Also, the REST Proxy only supports producing, so if you need to consume anything you'll need to use a Kafka client.

Confluent has also been pushing things toward Confluent Cloud, which currently does not include REST Proxy (despite what their salespeople may say). So if you tie yourself to their REST Proxy implementation in Confluent Platform you may find that you need to start managing your own instance.

Answer 2

One advantage of a native client would be raw performance via direct TCP to the brokers rather than round trip HTTP serialization + JVM serialization taking place within the REST Proxy.

A disadvantage with the above could be maintaining security policies for all your clients in various languages; otherwise anyone could produce and consume any one's other topics. If you don't have a multi-tenant Kafka cluster, maybe that's not a factor. Compared to the REST proxy, there's only one SSL cert, but having ACLs on topics might still be a good idea.

The obvious advantage of the REST proxy is that there's a standard interface for any HTTP client to transact with. There's no need to distinguish Kafka versions or supported APIs in clients besides what the proxy itself supports. Over time, though, this will become less and less needed as clients are being developed.

Also, your previous question What are the benefits of the Kafka REST Proxy API?

Answer 3

With the REST proxy, it's much easier to run admin tasks without having to use the lower level Kafka protocol. You can do things like view the state of the cluster, for example. It basically allows you to do the same thing as you would at a lower level but in an easier way.