CODEWITHSUNDEEP
nginxapache-kafka
qizer
qizer

Reputation: 559

Allow access to kafka via nginx

Good day, I want to connect to my kafka server from the internet. Kafka installed on the virtual server and all servers hidden behind a nginx.

I updated kafka settings (server.properties). Added: listeners=PLAINTEXT://:9092

I can connect to kafka server from local network via ip address 10.0.0.1:9092, but unable connect from internet by domain name.

Response from kafka: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic test-topic not present in metadata after 60000 ms.

Nginx: [26/Nov/2019:12:38:25 +0100] "\x00\x00\x00\x14\x00\x12\x00\x02\x00\x00\x00\x00\x00" 400 166 "-" "-" "request_time=1.535" "upstream_response_time=-" "upstream_connect_time=-" "upstream_header_time=-"

nginx conf:

server {
    listen       9092;
    server_name  site.name;

    # Max Request size
    client_max_body_size 20m;

    location / {
        proxy_pass      http://10.0.0.1:9092;
    }
}

Does anyone know what the problem is?

Upvotes: 2

Views: 14467

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191983

Kafka doesn't use http protocol for communication, so it can't be fronted by an HTTP reverse proxy.

You'll have to use nginx stream definition blocks for TCP proxying

(I've not tried this personally)

https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/

unable connect from internet by domain name.

Sounds like an issue with your advertised.listeners configuration. Note that there is no clear way to "hide" Kafka behind a proxy since your clients are required to communicate directly with each broker individually (therefore defeating the purpose of having Ngnix unless you want to use one Nginx server or open a new port, per broker), and would therefore also require Kafka to know that it would need to "advertise" the proxy rather than its own address.

If you really want to expose Kafka to the public web, you should really be using SSL/SASL listeners, not PLAINTEXT

If you want to use HTTP, then you can install Kafka REST Proxy, then put Nginx in front of that. Then your clients would use http rather than standard kafka libraries

Upvotes: 7

Related Questions