CODEWITHSUNDEEP
kubernetesgoogle-kubernetes-enginevpc
Mike Furlender
Mike Furlender

Reputation: 4019

Google Kubernetes private cluster: Google Compute Engine: Exceeded maximum supported number of secondary ranges per subnetwork

We are attempting to make several private Kubernetes clusters. We can find limited documentation on specific settings for the private cluster, therefore we are running into issues related to the subnetwork IP ranges.

Say we have 3 clusters: We set the Master Address Range to 172.16.0.0/28, 172.16.0.16/28 and 172.16.0.32/28 respectively.

We leave Network and Subnet set to "default". We are able to create 2 clusters that way, however, upon spin-up of the 3rd cluster, we receive the error of "Google Compute Engine: Exceeded maximum supported number of secondary ranges per subnetwork: 5." We suspect that we are setting up the subnetwork IP ranges incorrectly, but we are not sure what we are doing wrong, or why there is more than 1 secondary range per subnetwork, to begin with.

Here is a screenshot of the configuration for one of the clusters: kubernetes configuration screenshot

We are setting these clusters up through the UI.

Upvotes: 3

Views: 2527

Answers (6)

Hui Zheng
Hui Zheng

Reputation: 3087

I run into the same issue with a error message below

ERROR: (gcloud.container.clusters.create) ResponseError: code=400, message=This operation will exceed max secondary ranges per subnetwork (30) for subnet "default", consider reusing existing secondary ranges or use a different subnetwork.

I think the problem is we create GEK clusters that shares the same subnet "default" and eventually they exceed the max secondary ranges.

I think the best practice would be to create dedicated subnet for each cluster.

You Creating a cluster and subnet simultaneously

gcloud container clusters create cluster-name \
    --region=region \
    --enable-ip-alias \
    --create-subnetwork name=subnet-name

see below to find more subnet configuration https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips#creating_cluster_and_subnet

NOTE: GKE tries to clean up the created subnetwork when the cluster is deleted. But if the subnetwork is being used by other resources, GKE does not delete the subnetwork, and you must manage the life of the subnetwork yourself.

Upvotes: 0

Eyal Levin
Eyal Levin

Reputation: 18416

You can create clusters via gcloud and add --create-subnetwork "".

See: https://cloud.google.com/sdk/gcloud/reference/container/clusters/create#--create-subnetwork

This will create a new subnet with each new cluster so the "5 Secondary IP ranges per subnet" quota won't be reached.

Upvotes: 1

eurico
eurico

Reputation: 21

(Can't comment on Alan's answer due to low reputation)

You can create a new subnetwork:

  1. go to the "VPC network"
  2. click on "default" (under name)
  3. click on "Add subnet"
  4. define the subnet range / zone

Then on GKE when you create a new cluster, select your new subnetwork. This should allow you to create more clusters without running into the error Google Compute Engine: Exceeded maximum supported number of secondary ranges per subnetwork: 5.

Upvotes: 1

Necevil
Necevil

Reputation: 2912

For anyone who lands here from Google and is wondering how to list / see the subnet names that have been created using GKE as described in OP's question:

To list subnets for a region (and potentially modify or delete a Subnet, since you won't know the name) use the beta gcloud command:

gcloud beta container subnets list-usable

I landed here while looking for the answer and figured others trying to determine the best way to structure their subnets / ranges might be able to use the above command (which took me forever to track down).

Upvotes: 2

Alan Grosskurth
Alan Grosskurth

Reputation: 231

The best approach is to create a new subnetwork for each cluster. This way, each subnetwork only requires 2 secondary ranges, and you won't hit the limit of 5.

Upvotes: 1

user571470
user571470

Reputation: 404

This cluster has VPC-native (alias IP) enabled, which use 2 secondary ranges per cluster.

See https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips#secondary_ranges

According to

Google Compute Engine: Exceeded maximum supported number of secondary ranges per subnetwork: 5.

the max is 5. That's why the 3rd one failed to create.

Upvotes: 2

Related Questions