CODEWITHSUNDEEP
kubernetesprometheusbitnami
Dolphin
Dolphin

Reputation: 39005

0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports when start the promethus exporter

After I install the promethus using helm in kubernetes cluster, the pod shows error like this:

0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

this is the deployment yaml:

apiVersion: v1
kind: Pod
metadata:
  name: kube-prometheus-1660560589-node-exporter-n7rzg
  generateName: kube-prometheus-1660560589-node-exporter-
  namespace: reddwarf-monitor
  uid: 73986565-ccd8-421c-bcbb-33879437c4f3
  resourceVersion: '71494023'
  creationTimestamp: '2022-08-15T10:51:07Z'
  labels:
    app.kubernetes.io/instance: kube-prometheus-1660560589
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: node-exporter
    controller-revision-hash: 65c69f9b58
    helm.sh/chart: node-exporter-3.0.8
    pod-template-generation: '1'
  ownerReferences:
    - apiVersion: apps/v1
      kind: DaemonSet
      name: kube-prometheus-1660560589-node-exporter
      uid: 921f98b9-ccc9-4e84-b092-585865bca024
      controller: true
      blockOwnerDeletion: true
status:
  phase: Pending
  conditions:
    - type: PodScheduled
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2022-08-15T10:51:07Z'
      reason: Unschedulable
      message: >-
        0/1 nodes are available: 1 node(s) didn't have free ports for the
        requested pod ports.
  qosClass: BestEffort
spec:
  volumes:
    - name: proc
      hostPath:
        path: /proc
        type: ''
    - name: sys
      hostPath:
        path: /sys
        type: ''
    - name: kube-api-access-9fj8v
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: node-exporter
      image: docker.io/bitnami/node-exporter:1.3.1-debian-11-r23
      args:
        - '--path.procfs=/host/proc'
        - '--path.sysfs=/host/sys'
        - '--web.listen-address=0.0.0.0:9100'
        - >-
          --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
        - >-
          --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
      ports:
        - name: metrics
          hostPort: 9100
          containerPort: 9100
          protocol: TCP
      resources: {}
      volumeMounts:
        - name: proc
          readOnly: true
          mountPath: /host/proc
        - name: sys
          readOnly: true
          mountPath: /host/sys
        - name: kube-api-access-9fj8v
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      livenessProbe:
        httpGet:
          path: /
          port: metrics
          scheme: HTTP
        initialDelaySeconds: 120
        timeoutSeconds: 5
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 6
      readinessProbe:
        httpGet:
          path: /
          port: metrics
          scheme: HTTP
        initialDelaySeconds: 30
        timeoutSeconds: 5
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 6
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        runAsUser: 1001
        runAsNonRoot: true
  restartPolicy: Always
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: kube-prometheus-1660560589-node-exporter
  serviceAccount: kube-prometheus-1660560589-node-exporter
  hostNetwork: true
  hostPID: true
  securityContext:
    fsGroup: 1001
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchFields:
              - key: metadata.name
                operator: In
                values:
                  - k8smasterone
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app.kubernetes.io/instance: kube-prometheus-1660560589
                app.kubernetes.io/name: node-exporter
            namespaces:
              - reddwarf-monitor
            topologyKey: kubernetes.io/hostname
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
    - key: node.kubernetes.io/disk-pressure
      operator: Exists
      effect: NoSchedule
    - key: node.kubernetes.io/memory-pressure
      operator: Exists
      effect: NoSchedule
    - key: node.kubernetes.io/pid-pressure
      operator: Exists
      effect: NoSchedule
    - key: node.kubernetes.io/unschedulable
      operator: Exists
      effect: NoSchedule
    - key: node.kubernetes.io/network-unavailable
      operator: Exists
      effect: NoSchedule
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority

I have checked the host machine and found the port 9100 is free, why still told that no port for this pod? what should I do to avoid this problem? this is the host port 9100 check command:

[root@k8smasterone grafana]# lsof -i:9100
[root@k8smasterone grafana]#

this is the pod describe info:

➜  ~ kubectl describe pod kube-prometheus-1660560589-node-exporter-n7rzg -n reddwarf-monitor
Name:           kube-prometheus-1660560589-node-exporter-n7rzg
Namespace:      reddwarf-monitor
Priority:       0
Node:           <none>
Labels:         app.kubernetes.io/instance=kube-prometheus-1660560589
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=node-exporter
                controller-revision-hash=65c69f9b58
                helm.sh/chart=node-exporter-3.0.8
                pod-template-generation=1
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  DaemonSet/kube-prometheus-1660560589-node-exporter
Containers:
  node-exporter:
    Image:      docker.io/bitnami/node-exporter:1.3.1-debian-11-r23
    Port:       9100/TCP
    Host Port:  9100/TCP
    Args:
      --path.procfs=/host/proc
      --path.sysfs=/host/sys
      --web.listen-address=0.0.0.0:9100
      --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
      --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
    Liveness:     http-get http://:metrics/ delay=120s timeout=5s period=10s #success=1 #failure=6
    Readiness:    http-get http://:metrics/ delay=30s timeout=5s period=10s #success=1 #failure=6
    Environment:  <none>
    Mounts:
      /host/proc from proc (ro)
      /host/sys from sys (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9fj8v (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  proc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:
  sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:
  kube-api-access-9fj8v:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason            Age                      From               Message
  ----     ------            ----                     ----               -------
  Warning  FailedScheduling  2m54s (x233 over 3h53m)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

this is the netstat:

[root@k8smasterone ~]# netstat -plant |grep 9100
[root@k8smasterone ~]#

I also tried this to allow the pods running in master node by add this config:

tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master

still did not fixed this problem.

Upvotes: 10

Views: 29478

Answers (1)

Aqumik
Aqumik

Reputation: 1

I also encountered the same error when I tried to deploy multiple ingress-nginx pods on the same node.

My solution is setting hostNetwork: false and specify ip address.

values.yaml

controller:
  service:
    loadBalancerIP: 10.13.35.138
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
  electionID: ingress-nginx-private-test
  # hostNetwork: true
  replicaCount: 3

Upvotes: 0

Related Questions