Cannot install private dependency from artifact registry inside docker build

Question

I am trying to install a private python package that was uploaded to an artifact registry inside a docker container (to deploy it on cloudrun).

I have sucessfully used that package in a cloud function in the past, so I am sure the package works.

cloudbuild.yaml

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/${_PROJECT}/${_SERVICE_NAME}:$SHORT_SHA', '--network=cloudbuild', '.', '--progress=plain']

Dockerfile

FROM python:3.8.6-slim-buster

ENV APP_PATH=/usr/src/app
ENV PORT=8080

# Copy requirements.txt to the docker image and install packages
RUN apt-get update && apt-get install -y cython 

RUN pip install --upgrade pip

# Set the WORKDIR to be the folder
RUN mkdir -p $APP_PATH

COPY / $APP_PATH

WORKDIR $APP_PATH

RUN pip install -r requirements.txt --no-color
RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3 # This line is where the bug occurs


# Expose port 
EXPOSE $PORT

# Use gunicorn as the entrypoint
CMD exec gunicorn --bind 0.0.0.0:8080 app:app

The permissions I added are:

• cloudbuild default service account (project-number@cloudbuild.gserviceaccount.com): Artifact Registry Reader
• service account running the cloudbuild : Artifact Registry Reader
• service account running the app: Artifact Registry Reader

The cloudbuild error:

Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 340, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__

Answer 1

For future readers that are trying to get this to work on Cloudbuild, the --network=cloudbuild flag in cloudbuild binds your Application Default Credentials (ADC) to the build container, but doesn't authenticate your artifact registry. You'll need to do this manually.

You can leverage Cloudbuild gcloud builder and Docker's multi-stage to get the access token in one stage, and setup pip.conf and perform the regular pip install in the other stage. The solution aims to minimize the exposure of sensitive credentials as well.

Your cloudbuild step can remain unchanged and for your Dockerfile:

## First stage: Obtain gcloud access token
FROM gcr.io/cloud-builders/gcloud as gcloud

RUN gcloud auth print-access-token > /token.txt


## Second stage: Build application image
FROM python:3.8.6-slim-buster

# ... your other dockerfile commands

COPY --from=gcloud /token.txt /token.txt
COPY / $APP_PATH
WORKDIR $APP_PATH
RUN echo "[global]\nextra-index-url = https://oauth2accesstoken:$(cat /token.txt)@YOUR_PRIVATE_REPO_HOSTNAME" > /etc/pip.conf && \
    pip install -r requirements.txt && \
    pip install my-package==0.2.3 && \
    rm /etc/pip.conf /token.txt

# ... your other dockerfile commands

Replace YOUR_PRIVATE_REPO_HOSTNAME with your private repo hostname, i.e.

<region>-python.pkg.dev/<project_id>/<private_repo>/simple/

Answer 2

The best way of doing this is to mount a Docker secret on build.

To do it, you need to add this to your Dockerfile:

# Argument `GOOGLE_APPLICATION_CREDENTIALS` will be set from 
# Docker `/run/secrets/gsa_key` value
ARG GOOGLE_APPLICATION_CREDENTIALS=/run/secrets/gsa_key

# Install the keyring, needed to access Google Artifact Registry.
RUN pip install keyring keyrings.google-artifactregistry.auth

# Mount the secret (!) and install the private package.
RUN --mount=type=secret,id=gsa_key pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3

I.e., your Dockerifle at the end should look more or less like this:

FROM python:3.8.6-slim-buster

ENV APP_PATH=/usr/src/app
ENV PORT=8080

# Copy requirements.txt to the docker image and install packages
RUN apt-get update && apt-get install -y cython 

RUN pip install --upgrade pip

# Set the WORKDIR to be the folder
RUN mkdir -p $APP_PATH

COPY / $APP_PATH

WORKDIR $APP_PATH

# Argument `GOOGLE_APPLICATION_CREDENTIALS` will be set from 
# Docker `/run/secrets/gsa_key` value
ARG GOOGLE_APPLICATION_CREDENTIALS=/run/secrets/gsa_key

RUN pip install -r requirements.txt --no-color

# Install the keyring, needed to access Google Artifact Registry.
RUN pip install keyring keyrings.google-artifactregistry.auth

# Mount the secret (!) and install the private package.
RUN --mount=type=secret,id=gsa_key pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3

# Expose port 
EXPOSE $PORT

# Use gunicorn as the entrypoint
CMD exec gunicorn --bind 0.0.0.0:8080 app:app

And with that in place, you will build your Docker image with this command:

DOCKER_BUILDKIT=1 docker build \
  --secret id=gsa_key,src=/path/to/your/google/credentials.json \
  -t my-application .

Just adapt that to your cloudbuild.yaml and you should be fine. Also, notice that you don't need to do multistage as the secret is never shown in the Docker image.

P.D.: You will need a modern Docker version, and Docker Buildkit enabled.

Answer 3

From your traceback log, we can see that Cloud Build doesn't have the credentials to authenticate to the private repo:

Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception: //<-ASKING FOR USERNAME

I uploaded a simple package to a private Artifact Registry repo to test this out when building a container and also received the same message. Since you seem to be authenticating with a service account key, the username and password will need to be stored inside pip.conf:

pip.conf

[global]
extra-index-url = https://_json_key_base64:KEY@LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple/

This file therefore needs to be available during the build process. Multi-stage docker builds are very useful here to ensure the configuration keys are not exposed, since we can choose what files make it into the final image (configuration keys would only be present while used to download the packages from the private repo):

Sample Dockerfile

# Installing packages in a separate image
FROM python:3.8.6-slim-buster as pkg-build

# Target Python environment variable to bind to pip.conf
ENV PIP_CONFIG_FILE /pip.conf

WORKDIR /packages/
COPY requirements.txt /

# Copying the pip.conf key file only during package downloading
COPY ./config/pip.conf /pip.conf

# Packages are downloaded to the /packages/ directory
RUN pip download -r /requirements.txt
RUN pip download --extra-index-url https://LOCATION-python.pkg.dev/PROJECT/REPO/simple/ PACKAGES

# Final image that will be deployed
FROM python:3.8.6-slim-buster

ENV PYTHONUNBUFFERED True
ENV APP_HOME /app

WORKDIR /packages/
# Copying ONLY the packages from the previous build
COPY --from=pkg-build /packages/ /packages/

# Installing the packages from the copied files
RUN pip install --no-index --find-links=/packages/ /packages/*

WORKDIR $APP_HOME
COPY ./src/main.py ./

# Executing sample flask web app 
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app

I based the dockerfile above on this related thread, and I could confirm the packages were correctly downloaded from my private Artifact Registry repo, and also that the pip.conf file was not present in the resulting image.