CODEWITHSUNDEEP
kubernetesdocker-registryhaproxy-ingress
echappy
echappy

Reputation: 543

Issue Accessing Docker Registry from Inside Kubernetes

I have setup a private docker registry inside my Kubernetes Cluster. The deployment is as follows

apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
  labels:
    app: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
    spec:
      volumes:
        - name: auth-vol
          secret:
            secretName: "registry-credentials"

      containers:
        - image: registry:2
          name: registry
          imagePullPolicy: IfNotPresent
          env:
            - name: REGISTRY_AUTH
              value: "htpasswd"
            - name: REGISTRY_AUTH_HTPASSWD_REALM
              value: "k8s_user"
            - name: REGISTRY_AUTH_HTPASSWD_PATH
              value: "/auth/htpasswd"
          ports:
            - containerPort: 5000
          volumeMounts:
            - name: auth-vol
              mountPath: /auth

I am routing using the following Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: registry-ingress
spec:
  rules:
    - host: "registry.<my domain>"
      http:
        paths:
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: registry
                port:
                  number: 80

Externally I have a load balancer terminating SSL and then forwarding the request to the appropriate ingress port for HTTP traffic. From outside the network, I have no problems pushing/pulling from the registry. However from inside the network, I am getting the following error when I try and deploy something and run kubectl pod describe <pod>

  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  26s                default-scheduler  Successfully assigned default/server-6df575c99c-ltwqr to k8s-root-default-pool-3de67
  Normal   BackOff    24s (x2 over 25s)  kubelet            Back-off pulling image "registry.<mydomain>/server:0.0.1"
  Warning  Failed     24s (x2 over 25s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    11s (x2 over 25s)  kubelet            Pulling image "registry.<mydomain>/server:0.0.1"
  Warning  Failed     11s (x2 over 25s)  kubelet            Failed to pull image "registry.<mydomain>/server:0.0.1": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.<mydomain>/v2/: x509: certificate is valid for haproxy-controller.default, not registry.<mydomain>.io
  Warning  Failed     11s (x2 over 25s)  kubelet            Error: ErrImagePull

It appears as though the request is hitting the HAProxy Ingress controller certificate rather than going to the outside world and hitting the load balancer's SSL certificate. Is there some better way I should be doing this?

Upvotes: 0

Views: 373

Answers (1)

echappy
echappy

Reputation: 543

I figured this out. Before I was using kubectl expose deployment/registry to automatically create the service. I figured out that if I create a NodePort service, this will expose it on a fixed port on all nodes

apiVersion: v1
kind: Service
metadata:
  name: registry
spec:
  type: NodePort
  selector:
    app: registry
  ports:
    - port: 5000
      targetPort: 5000
      nodePort: 32500

This then allowed me to use "localhost:32500" to access the registry on all nodes. I also had to update my deployment to pull the image from "localhost:32500".

Upvotes: 1

Related Questions