CODEWITHSUNDEEP
apache-kafka
dm90
dm90

Reputation: 795

Ingress for kafka

We are exploring in implementing the multi-tenancy at kafka for each of our dev team which would be hosted in AWS-EKS.

  1. For this the initial thought process is to have topic level multi-tenant.
  2. NLB-Nginx-Ingress: ingress host-route for each team and add all the brokers in the backend, in which that team's all the topic-partition leaders are present.
  3. access restriction through the ACLs at broker level based on principal like user.

Sample flow:

enter image description here

Ingress book-keeping challenges:

  1. When someone from foobar team creates a new topic and if that lands in a new broker, we need to add that broker to the backend of the respective ingress.
  2. If a broker goes down, again the ingress need to be updated.
  3. Prune the brokers when the partition leader goes away due to topic deletion.

What I'm Looking for: Apart from writing an operator or app to do the above tasks, is there any other better way to achieve this? I'm ok to completely new suggestions as well. Since this is just in the POC stage.

PS: I'm new to kafka and if this exchange is not suitable for this question, pls suggest the right exchange to post. Thanks!

Upvotes: 0

Views: 870

Answers (1)

Ran Lupovich
Ran Lupovich

Reputation: 1841

First of all the ACL restrictions are cluster level and not broker level,

Secondly, for bootstraping process you need to access at least one active broker from the cluster it will send back metadata where the data leaders are and on the continuous connection the client will connect to the brokers accordingly,

there is no need to put load balancer behind kafka bootstraping , the suggestion is to put at least two brokers or more in comma separated list , the client will connect the first available and get the metadata, for further connection , client need to be able to connect to all brokers in the cluster

You can use the ACL to restrict access by principals (users) to topics in the cluster based on their need.

Upvotes: 1

Related Questions