CODEWITHSUNDEEP
dockergitlab-ci
cclloyd
cclloyd

Reputation: 9195

Gitlab CI Split Docker Build Into Multiple Stages

I have a react/django app that's dockerized. There's 2 stages to the GitLab CI process. Build_Node and Build_Image. Build node just builds the react app and stores it as an artifact. Build image runs docker build to build the actual docker image, and relies on the node step because it copies the built files into the image.

However, the build process on the image takes a long time if package dependencies have changed (apt or pip), because it has to reinstall everything.

Is there a way to split the docker build job into multiple parts, so that I can say install the apt and pip packages in the dockerfile while build_node is still running, then finish the docker build once that stage is done?

gitlab-ci.yml:

stages:
  - Build Node Frontend
  - Build Docker Image

services:
  - docker:18.03-dind

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""

build_node:
  stage: Build Node Frontend
  only:
    - staging
    - production
  image: node:14.8.0
  variables:
    GIT_SUBMODULE_STRATEGY: recursive
  artifacts:
    paths:
      - http
  cache:
    key: "node_modules"
    paths:
      - frontend/node_modules
  script:
    - cd frontend
    - yarn install --network-timeout 100000
    - CI=false yarn build
    - mv build ../http



build_image:
  stage: Build Docker Image
  only:
    - staging
    - production
  image: docker

  script:
    #- sleep 10000
    - tar -cvf app.tar api/ discordbot/ helpers/ http/
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    #- docker pull $CI_REGISTRY_IMAGE:latest
    #- docker build --network=host --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA --tag $CI_REGISTRY_IMAGE:latest .
    - docker build --network=host --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA --tag $CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest

Dockerfile:

FROM python:3.7-slim

# Add user
ARG APP_USER=abc
RUN groupadd -r ${APP_USER} && useradd --no-log-init -r -g ${APP_USER} ${APP_USER}

WORKDIR /app
ENV PYTHONUNBUFFERED=1
EXPOSE 80
EXPOSE 8080

ADD requirements.txt /app/

RUN set -ex \
    && BUILD_DEPS=" \
        gcc \
    " \
    && RUN_DEPS=" \
        ffmpeg \
        postgresql-client \
        nginx \
        dumb-init \
    " \
    && apt-get update && apt-get install -y $BUILD_DEPS \
    && pip install --no-cache-dir --default-timeout=100000 -r /app/requirements.txt \
    && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false  $BUILD_DEPS \
    && apt-get update && apt-get install -y --no-install-recommends $RUN_DEPS \
    && rm -rf /var/lib/apt/lists/*

# Set uWSGI settings
ENV UWSGI_WSGI_FILE=/app/api/api/wsgi.py UWSGI_HTTP=:8000 UWSGI_MASTER=1 UWSGI_HTTP_AUTO_CHUNKED=1 UWSGI_HTTP_KEEPALIVE=1 UWSGI_LAZY_APPS=1 UWSGI_WSGI_ENV_BEHAVIOR=holy PYTHONUNBUFFERED=1 UWSGI_WORKERS=2 UWSGI_THREADS=4
ENV UWSGI_STATIC_EXPIRES_URI="/static/.*\.[a-f0-9]{12,}\.(css|js|png|jpg|jpeg|gif|ico|woff|ttf|otf|svg|scss|map|txt) 315360000"
ENV PYTHONPATH=$PYTHONPATH:/app/api:/app
ENV DB_PORT=5432 DB_NAME=shittywizard DB_USER=shittywizard DB_HOST=localhost

ADD nginx.conf /etc/nginx/nginx.conf

# Set entrypoint
ADD entrypoint.sh /
RUN chmod 755 /entrypoint.sh
ENTRYPOINT ["dumb-init", "--", "/entrypoint.sh"]

ADD app.tar /app/

RUN python /app/api/manage.py collectstatic --noinput

Upvotes: 2

Views: 7737

Answers (1)

Michael Delgado
Michael Delgado

Reputation: 15432

Sure! Check out the gitlab docs on stages and on building docker images with gitlab-ci.

If you have multiple pipeline steps defined within a stage they will run in parallel. For example, the following pipeline would build the node and image artifacts in parallel and then build the final image using both artifacts.

stages:
- build
- bundle

build-node:
  stage: build
  script:
  - # steps to build node and push to artifact registry

build-base-image:
  stage: build
  script:
  - # steps to build image and push to artifact registry

bundle-node-in-image:
  stage: bundle
  script:
  - # pull image artifact
  - # download node artifact
  - # build image on top of base image with node artifacts embedded

Note that all the pushing and pulling and starting and stopping might not save you time depending on your image size relative to build time, but this will do what you're asking for.

Upvotes: 4

Related Questions