How to wait until Kubernetes assigned an external IP to a LoadBalancer service?

Question

Creating a Kubernetes LoadBalancer returns immediatly (ex: kubectl create -f ... or kubectl expose svc NAME --name=load-balancer --port=80 --type=LoadBalancer).

I know a manual way to wait in shell:

external_ip=""
while [ -z $external_ip ]; do
    sleep 10
    external_ip=$(kubectl get svc load-balancer --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
done

This is however not ideal:

• Requires at least 5 lines Bash script.
• Infinite wait even in case of error (else requires a timeout which increases a lot line count).
• Probably not efficient; could use --wait or --wait-once but using those the command never returns.

Is there a better way to wait until a service external IP (aka LoadBalancer Ingress IP) is set or failed to set?

Answer 1

We had a similar problem on AWS EKS and wanted to have a one-liner for that to use in our CI pipelines. We simply "watched" the output of a command until a particular string is observed and then exit using the until loop:

until kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done

To avoid an infinite loop you could enhance it using timeout (brew install coreutils on a Mac):

timeout 10s bash -c 'until kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done'

Getting the ip after that is easy using:

kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer.ingress[0].ip}' 

or when using a service like AWS EKS you most likely have hostname populated instead of ip:

kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}' 

Sidenote (Update 03.2023): kubectl wait would be ideal, but will not be able to wait on arbitrary jsonpath until v1.23 (see this PR). BUT even from 1.24 on we seem to not be able to use kubectl wait since .status.loadBalancer.ingress[0] is a list, which is not supported by the kubectl wait jsonpath implementation. So the command kubectl wait service/<service-name> --for=jsonpath='{.status.loadBalancer}'=ingress throws one of the following errors (see also this so question):

error: jsonpath wait format must be --for=jsonpath='{.status.readyReplicas}'=3
error: jsonpath leads to a nested object or list which is not supported

Answer 2

Maybe this is not the solution that you're looking for but at least it has less lines of code:

until [ -n "$(kubectl get svc load-balancer -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do
    sleep 10
done

Answer 3

This is little bit tricky by working solution:

kubectl get service -w load-balancer -o 'go-template={{with .status.loadBalancer.ingress}}{{range .}}{{.ip}}{{"\n"}}{{end}}{{.err}}{{end}}' 2>/dev/null | head -n1

Answer 4

Here's a generic bash function to watch with timeout, for any regexp in the output of a given command:

function watch_for() {
  CMD="$1" # Command to watch. Variables should be escaped \$
  REGEX="$2" # Pattern to search
  ATTEMPTS=${3:-10} # Timeout. Default is 10 attempts (interval of second)
  COUNT=0;

  echo -e "# Watching for /$REGEX/ during $ATTEMPTS seconds, on the output of command:\n# $CMD"
  until eval "$CMD" | grep -m 1 "$REGEX" || [[ $COUNT -eq $ATTEMPTS ]]; do
    echo -e "$(( COUNT++ ))... \c"
    sleep 1
  done
  if [[ $COUNT -eq $ATTEMPTS ]]; then
    echo "# Limit of $ATTEMPTS attempts has exceeded."
    return 1
  fi
  return 0
}

And here's how I used it to wait until a worker node gets an external IP (which took more than a minute):

$ watch_for "kubectl get nodes -l node-role.kubernetes.io/worker -o wide | awk '{print \$7}'" \
"[0-9]" 100

0... 1... 2... 3... .... 63... 64... 3.22.37.41

Answer 5

Really just a clean-up of @Dan Garfield's working example; My OCD wouldn't let this slide. In this case:

• on GCP
• requesting an internal lb
• with an annotation in a service definition

---

apiVersion: v1
kind: Service
metadata:
  name: yo
  annotations:
    cloud.google.com/load-balancer-type: "Internal"
    # external-dns.alpha.kubernetes.io/hostname: vault.stage.domain.tld.
...

NOTE: I've only been able to get external-dns to associate names to public IP addresses.

---

This has been scripted to accept a few arguments, now it's a library; example:

myServiceLB=$1
while true; do                                                                     
    successCond="$(kubectl get svc "$myServiceLB" \                                
        --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")"        
    if [[ -z "$successCond" ]]; then                                               
        echo "Waiting for endpoint readiness..."                                   
        sleep 10                                                                   
    else                                                                           
        sleep 2                                                                    
        export lbIngAdd="$successCond"                                             
        pMsg """
            The Internal LoadBalancer is up!
        """                                                                        
        break                                                                      
    fi                                                                             
done

Later, $lbIngAdd can be used to set records. Seems like -o jsonpath="{.status.loadBalancer.ingress[*].ip}" would work as well; whatever works.

Thanks for getting us started Dan :-)

Answer 6

Just to add to the answers here, the best option right now is to use a bash script. For convenience, I've put it into a single line that includes exporting an environmental variable.

Command to wait and find Kubernetes service endpoint

bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc NAME_OF_YOUR_SERVICE --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; export endpoint=$external_ip'

I've also modified your script so it only executes a wait if the ip isn't available. The last bit will export an environment variable called "endpoint"

Bash Script to Check a Given Service

Save this as check-endpoint.sh and then you can execute $sh check-endpoint.sh SERVICE_NAME

#!/bin/bash
# Pass the name of a service to check ie: sh check-endpoint.sh staging-voting-app-vote
# Will run forever...
external_ip=""
while [ -z $external_ip ]; do
  echo "Waiting for end point..."
  external_ip=$(kubectl get svc $1 --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
  [ -z "$external_ip" ] && sleep 10
done
echo 'End point ready:' && echo $external_ip

Using this in a Codefresh Step

I'm using this for a Codefresh pipeline and it passes a variable $endpoint when it's done.

  GrabEndPoint:
    title: Waiting for endpoint to be ready
    image: codefresh/plugin-helm:2.8.0
    commands:
      - bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc staging-voting-app-vote --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; cf_export endpoint=$external_ip'

Answer 7

There's not really a "failed to set" condition because we will retry it forever. A failure might have been a transient error in the cloud provider or a quota issue that gets resolved over the course of hours or days, or any number of things. The only failure comes from "how long are you willing to wait?" - which only you can know.

We don't have a general "wait for expression" command because it ends up being arbitrarily complex and you're better off just coding that in a real language. Ergo the bash loop above. We could do better about having a 'watch' command, but it's still a timeout in the end.