Reputation: 49814
SSL handshake failed using Ingress way for Kafka created by Strimzi
I have a local k3s Kubernetes cluster created by multipass.
I am trying to set up Kafka using Ingress way based on this tutorial to make it accessible for clients running outside of Kubernetes.
Here are my steps:
First get my cluster IP by
➜ kubectl get nodes
NAME STATUS ROLES AGE VERSION
west-master Ready control-plane,master 15m v1.26.3+k3s1
➜ kubectl get node west-master -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
west-master Ready control-plane,master 16m v1.26.3+k3s1 192.168.205.5 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.19-k3s1
➜ kubectl cluster-info
Kubernetes control plane is running at https://192.168.205.5:6443
CoreDNS is running at https://192.168.205.5:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://192.168.205.5:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
which is 192.168.205.5 (Is this correct IP address for the one I used in my-kafka-persistent.yaml I used in next step?).
Then I deploy my Kafka by:
kubectl create namespace hm-kafka
kubectl apply --filename="https://strimzi.io/install/latest?namespace=hm-kafka" --namespace=hm-kafka
kubectl apply --filename=my-kafka-persistent.yaml --namespace=hm-kafka
my-kafka-persistent.yaml (based on kafka-persistent.yaml):
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: hm-kafka
spec:
kafka:
version: 3.4.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
- name: external
port: 9094
type: ingress
tls: true
configuration:
bootstrap:
host: kafka-bootstrap.192.168.205.5.nip.io
brokers:
- broker: 0
host: kafka-broker-0.192.168.205.5.nip.io
- broker: 1
host: kafka-broker-1.192.168.205.5.nip.io
- broker: 2
host: kafka-broker-2.192.168.205.5.nip.io
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
inter.broker.protocol.version: "3.4"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi
deleteClaim: false
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 100Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}
After deployment:
Pods
Services
Ingresses
Also, for each Ingress, I can see SSL passthrough in the annotations:
Then I succeed following the tutorial creating the truststore.
➜ kubectl get secret hm-kafka-cluster-ca-cert \
--namespace=hm-kafka \
--output=jsonpath="{.data.ca\.crt}" \
| base64 -d \
> ca.crt
➜ keytool -importcert \
-trustcacerts \
-alias root \
-file ca.crt \
-keystore kafka-truststore.jks \
-storepass my_passw0rd \
-noprompt
Certificate was added to keystore
However, when I try to produce data, I met this issue:
➜ kafka-console-producer \
--broker-list kafka-bootstrap.192.168.205.5.nip.io:443 \
--producer-property security.protocol=SSL \
--producer-property ssl.truststore.password=my_passw0rd \
--producer-property ssl.truststore.location=kafka-truststore.jks \
--topic my-topic
>[2023-04-14 15:57:06,047] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.192.168.205.5.nip.io/192.168.205.5:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:57:06,047] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.192.168.205.5.nip.io:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:57:06,200] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.192.168.205.5.nip.io/192.168.205.5:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:57:06,201] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.192.168.205.5.nip.io:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:57:06,691] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.192.168.205.5.nip.io/192.168.205.5:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:57:06,691] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.192.168.205.5.nip.io:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
Any guide would be appreciate, thanks!
UPDATE 1
Thanks @OneCricketeer pointing the issue!
As I am using multipass on macOS, I can provide INSTALL_K3S_EXEC="server --disable traefik", so the updated command to create k3s cluster is:
multipass launch --name=west-master --cpus=4 --memory=16g --disk=128g
multipass exec west-master -- \
bash -c 'curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --disable traefik" K3S_KUBECONFIG_MODE="644" sh -'
I actually switched to Rancher Desktop as it is also using k3s and easy to disable Traefik which can be set up in the UI.
UPDATE 2
Regarding how to deploy ingress-nginx and how to resolve another issue "ingress does not contain a valid IngressClass" I met, I posted at Strimzi Kafka brokers not be created because of "ingress does not contain a valid IngressClass"
Upvotes: 2
Views: 1180
Answers (1)
Reputation: 191738
k3s uses traefik, not nginx, so those annotations aren't doing anything... The referenced blog assumes you are using nginx instead
Restart your k3s cluster, but provide --no-deploy-traefik option, and install nginx ingress controller
Otherwise, you will need to refer to Traefik ingress docs on what matching annotations it will use for SSL passthrough.
Keeping in mind, Kafka is not an HTTP/S service, so you should not be using ports 80/443 to communicate with it.
Upvotes: 2