CODEWITHSUNDEEP
kubernetesgoogle-cloud-platformgoogle-kubernetes-engineautopilot
V Adhi Pragantha
V Adhi Pragantha

Reputation: 74

GKE autopilot with shared vpc ip exhausted

I have setup a new subnet in my shared VPC for GKE autopilot as the following:

node ip: 10.11.1.0/24
first secondary ip: 10.11.2.0/24
second secondary ip: 10.11.3.0/24

I tried to test it by running simple nginx images with 30 replicas.

based on my understanding:

I have 256 possible node IP
I have 256 possible pod IP
I have 256 possible service IP

after deploying, somehow my k8s are stuck with only 2 pods deployed and running. the rest is just in pending state with error code:IP_SPACE_EXHAUSTED

my question is how come? I still have plenty IP address, this is fresh deployed kubernetes cluster.

Upvotes: 0

Views: 1385

Answers (3)

VSZ
VSZ

Reputation: 11

I work on GKE Autopilot and would like to share a few suggestions and recent updates:

  • Assuming we’re talking about a shared VPC type environment where private IPs (10.0.0.0/x) are scarce, we strongly recommend using non-RFC 1918 IP ranges wherever possible. This blog post provides an overview of this strategy and example configurations for GKE Autopilot. Considerations must be made for node, service, and pod IPs. In particular, we suggest using the E Class (240.0.0.0/4) which provides 250M IPs.

  • Note that by default Autopilot automatically provides a /20 non-RFC range for services, supporting up to 4000 nodes.

  • Use IP Masquerading to reach VPC services from pods which use non-RFC IPs.

  • With version 1.28+, GKE Autopilot has moved from a static pods/nodes (and therefore IPs/node) to a dynamic model, where based on the size of the node, we allocate a proportional number of IP addresses. This reduces waste in cases where small nodes get the same number of IPs as large nodes.

Upvotes: 1

Reid123
Reid123

Reputation: 274

Pod CIDR ranges in Autopilot clusters

The default settings for Autopilot cluster CIDR sizes are as follows:

  • Subnetwork range: /23
  • Secondary IP address range for Pods: /17
  • Secondary IP address range for Services: /22

Autopilot has a maximum Pods per node of 32, you may check this link.

Autopilot cluster maximum number of nodes is pre-configured and immutable, you may check this link.

Upvotes: 1

Gari Singh
Gari Singh

Reputation: 12033

Autopilot sets "max pods per node" to 32. This results in a /26 (64 IP addresses) being assigned to each Autopilot node from the Pod secondary IP range. Since your Pod range is a /24, this means your Autopilot cluster can support a max of 4 nodes.

By default, Autopilot clusters start with 2 nodes (one runs some system stuff). Looks like your pods did not fit on either of these nodes, so Autopilot provisioned new nodes as required. Generally, Autopilot tries to find the best fit node sizes for your deployments and in this case looks like you ended up with a pod per node.

I'd recommend a /17 or a /16 for your Pod range to maximize the number of nodes.

Upvotes: 1

Related Questions