CODEWITHSUNDEEP
kubernetestcpnginx-ingress
winedolphin
winedolphin

Reputation: 1

nginx-ingress tcp services - connection refused

I'm currently deploying a new kubernetes cluster and I want to expose a mongodb service from outside the cluster using an nginx-ingress. I know that nginx-ingress is usually for layer 7 applications but also capable to work on layer 4 (TCP/UDP) according to the official documentation.

https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

My mongodb service is a ClusterIP serivce which is accssible on port 11717 (internal namespace):

kubectl get svc -n internal

mongodb         ClusterIP   10.97.63.154    <none>        11717/TCP                        3d20h

telnet 10.97.63.154 11717
Trying 10.97.63.154...
Connected to 10.97.63.154.

I literally tried every possible combination to achieve this goal but with no success. I'm using the nginx-ingress helm chart (daemonset type).

My nginx-ingress/templates/controller-daemonset.yaml file:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-nginx-ingress
  namespace: default
  labels:
    app.kubernetes.io/name: nginx-ingress-nginx-ingress
    helm.sh/chart: nginx-ingress-0.13.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: nginx-ingress
spec:
  selector:
    matchLabels:
      app: nginx-ingress-nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress-nginx-ingress
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9113"
        prometheus.io/scheme: "http"
    spec:
      serviceAccountName: nginx-ingress-nginx-ingress
      terminationGracePeriodSeconds: 30
      hostNetwork: false
      containers:
      - name: nginx-ingress-nginx-ingress
        image: "nginx/nginx-ingress:2.2.0"
        imagePullPolicy: "IfNotPresent"
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443
        - name: mongodb
          containerPort: 11717
          hostPort: 11717

        - name: prometheus
          containerPort: 9113
        - name: readiness-port
          containerPort: 8081
        readinessProbe:
          httpGet:
            path: /nginx-ready
            port: readiness-port
          periodSeconds: 1
        securityContext:
          allowPrivilegeEscalation: true
          runAsUser: 101 #nginx
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        resources:
          {}
        args:
          - /nginx-ingress-controller
          - -nginx-plus=false
          - -nginx-reload-timeout=60000
          - -enable-app-protect=false
          - -tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
          - -publish-service=$(POD_NAMESPACE)/ingress-nginx
          - -annotations-prefix=nginx.ingress.kubernetes.io
          - -enable-app-protect-dos=false
          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-ingress-nginx-ingress
          - -default-server-tls-secret=$(POD_NAMESPACE)/nginx-ingress-nginx-ingress-default-server-tls
          - -ingress-class=nginx
          - -health-status=false
          - -health-status-uri=/nginx-health
          - -nginx-debug=false
          - -v=1
          - -nginx-status=true
          - -nginx-status-port=8080
          - -nginx-status-allow-cidrs=127.0.0.1
          - -report-ingress-status
          - -external-service=nginx-ingress-nginx-ingress
          - -enable-leader-election=true
          - -leader-election-lock-name=nginx-ingress-nginx-ingress-leader-election
          - -enable-prometheus-metrics=true
          - -prometheus-metrics-listen-port=9113
          - -prometheus-tls-secret=
          - -enable-custom-resources=true
          - -enable-snippets=false
          - -enable-tls-passthrough=false
          - -enable-preview-policies=false
          - -enable-cert-manager=false
          - -enable-oidc=false
          - -ready-status=true
          - -ready-status-port=8081
          - -enable-latency-metrics=false

My nginx-ingress/templates/controller-service.yaml file:

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-nginx-ingress
  namespace: default
  labels:
    app.kubernetes.io/name: nginx-ingress-nginx-ingress
    helm.sh/chart: nginx-ingress-0.13.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: nginx-ingress
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  - name: mongodb
    port: 11717
    targetPort: 11717
    protocol: TCP
  selector:
    app:  nginx-ingress-nginx-ingress

My nginx-ingress/templates/tcp-services.yaml file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: default
data:
  "11717": internal/mongodb:11717

kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-ingress-nginx-ingress-d5vms   1/1     Running   0          61m
nginx-ingress-nginx-ingress-kcs4p   1/1     Running   0          61m
nginx-ingress-nginx-ingress-mnnn2   1/1     Running   0          61m



kubectl get svc -o wide
NAME                          TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                      AGE    SELECTOR
kubernetes                    ClusterIP      10.96.0.1       <none>        443/TCP                                      4d1h   <none>
nginx-ingress-nginx-ingress   LoadBalancer   10.99.176.220   <pending>     80:31700/TCP,443:31339/TCP,11717:31048/TCP   61m    app=nginx-ingress-nginx-ingress


telnet 10.99.176.220 80
Trying 10.99.176.220...
Connected to 10.99.176.220.
Escape character is '^]'.


telnet 10.99.176.220 11717
Trying 10.99.176.220...
telnet: Unable to connect to remote host: Connection refused

I can't understand why the connection is getting refused on port 11717.

How can I achieve this scenario:

mongo.myExternalDomain:11717 --> nginx-ingress service --> nginx-ingress pod --> mongodb service --> mongodb pod

Thanks in advance! I would appreciate any kind of help!

Upvotes: 0

Views: 1479

Answers (1)

sobczak.dev
sobczak.dev

Reputation: 1308

I had similar issue. Maybe this will help you. In my case it was in tcp-services configmap:

Shortly. Instead of this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: default
data:
  "11717": internal/mongodb:11717

please change to:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: default
data:
  "11717": internal/mongodb:11717:PROXY

Details:

  1. Edit the 'tcp-services' configmap to add a tcp .service 8000: namespace/service:8000.
  2. edit the nginx-controller service to add a port (port:8000 --> targetPort:8000) for the tcp service in step1
  3. Check /etc/nginx/nginx.conf in nginx controller pod and confirm it contains a 'server' block with correct listen 8000; directive for the tcp/8000 service.
  4. Edit the 'tcp-services' configmap again to add the proxy-protocol decode directive and now the k/v for the tcp/8000 service becomes 8000: namespace/service:8000:PROXY
  5. Check /etc/nginx/nginx.conf in nginx controller pod and there isn't any change comparing that from step3, it is still listen 8000;
  6. Edit some ingress rule (make some change like updating the host)
  7. Check /etc/nginx/nginx.conf in nginx controller pod again and now the listen directive for the tcp/8000 service becomes listen 8000 proxy_protocol; which is correct.

Upvotes: 0

Related Questions