How to add an configure GKE node pool access scope

Question

Running gcloud container clusters describe [CLUSTER NAME]: the scopes listed under the oauthScopes property does not include https://www.googleapis.com/auth/devstorage.read_only, but i need it to pull my private image from container registry.

How do i add to the auth scope

Edit: I am automating my deployment with Ansible playbooks

Answer 1

According to google cloud docs, we can set scopes for already created Compute Engine VM/GKE clusters as well.

Answer 2

According to google cloud docs on associating a service account to an instance

When you create an instance using the gcloud command-line tool or the Google Cloud Console, you can specify which service account the instance uses when calling Google Cloud APIs. The instance is automatically configured with the following access scopes:

• Read-only access to Cloud Storage: https://www.googleapis.com/auth/devstorage.read_only

• Write access to write Compute Engine logs:

  https://www.googleapis.com/auth/logging.write

• Write access to publish metric data to your Google Cloud projects: https://www.googleapis.com/auth/monitoring.write

• Read-only access to Service Management features required for Google Cloud Endpoints(Alpha): https://www.googleapis.com/auth/service.management.readonly

• Read/write access to Service Control features required for Google Cloud Endpoints(Alpha):

  https://www.googleapis.com/auth/servicecontrol

Cloud Storage is the permission I need, so it's expected to be enabled by default, but one thing to note is that that the above applies when an instance is created using the gcloud command-line tool or the Google Cloud Console

In my case I was creating my instance using Ansible playbook. google.cloud.gcp_container_node_pool which is the module I use in creating the node pool takes a couple of parameters which includes config then oauth_scopes which is:

The set of Google API scopes to be made available on all of the node VMs under the "default" service account. The following scopes are recommended, but not required, and by default are not included: https://www.googleapis.com/auth/compute is required for mounting persistent storage on your nodes. https://www.googleapis.com/auth/devstorage.read_only is required for communicating with gcr.io (the Google Container Registry). If unspecified, no scopes are added, unless Cloud Logging or Cloud Monitoring are enabled, in which case their required scopes will be added.

This means unlike when using the command-line tool or cloud console, when you create a node pool using the Ansible module no scopes are added by default. Except the scopes for cloud logging and monitoring which are always added if enabled for your project.

To fix this,I included a list of oath_scopes I would like to enable:

- name: create k8s node pool
  google.cloud.gcp_container_node_pool:
    name: "node-pool-{{ cluster_name }}"
    initial_node_count: "{{ initial_node_count }}"
    cluster: "{{ cluster }}"
    config:
      disk_size_gb: "{{ disk_size_gb }}"
      disk_type: "{{ disk_type }}"
      machine_type: "{{ machine_type }}"
      oauth_scopes: 
        - https://www.googleapis.com/auth/devstorage.read_only
        - https://www.googleapis.com/auth/logging.write
        - https://www.googleapis.com/auth/monitoring.write
    location: "{{ zone }}"
    project: "{{ project_id }}"
    auth_kind: serviceaccount
    service_account_file: "{{ credentials_file }}"
    state: present

When you run the playbook again, the node pool would be recreated, this time with the scopes you have specified.

Answer 3

In general, for most Google Cloud service accounts, configuring access to a registry only requires granting the appropriate IAM permissions.

1. Google Kubernetes Engine uses the service account configured on the VM instances of cluster nodes to push and pull images. You must grant the service account the appropriate permissions to access the storage bucket used by Container Registry. You can find appropriate permissions in the documentation. The minimum permission to download images from GCR is "Storage Viewer".

2. If your Google Kubernetes Engine uses the default Service Account, you need to additionally configure the storage access scope. To only pull private Docker images, the VM instance needs the read-only storage access scope.

To update a GKE cluster access scope to add new scopes, we can simply create a new node pool with the new scopes like this:

    gcloud container node-pools create ADJUSTED-SCOPES \
       --cluster <YOUR_CLUSTER_NAME> --zone <YOUR_ZONE> \
       --num-nodes 3 \
       --scopes https://www.googleapis.com/auth/devstorage.read_only

If you want to add multiple scopes. It can be specified, separated by commas. For example:

    gcloud container node-pools create ADJUSTED-SCOPES \
       --cluster <YOUR_CLUSTER_NAME> --zone <YOUR_ZONE> \
       --num-nodes 3 \
       --scopes https://www.googleapis.com/auth/devstorage.read_write,https://www.googleapis.com/WHAT_YOU_NEED

3. If you want to use the legacy access scopes in clusters running Kubernetes version 1.10 and higher, you must add the scopes manually while creating the cluster. Refer Migrating from legacy access scopes.

Refer required permissions and updating google kubernetes VM scopes for information.