GitLab CI bump Python package version

Question

I was wondering whether is possible to bump Python package version stored in GitLab inside GitLab CI runner.

I have example package structure:

/package
  /src
    /__init__.py
     main.py
  setup.py
  Dockerfile
  .gitlab-ci.yml

__init__.py includes:

  __version__ = '1.0.0'

setup.py includes:

  setup(
        name='foo',
        version=src.__version__,
        packages=find_packages(),
        install_required=[foo, bar]
  )

Simple workflow for bumping and releasing looks like here: Best workflow and practices for releasing a new python package version on github and pypi

But can we automatically bump version in __init__.py while releasing directly in GitLab CI?

Answer 1

An option is to use setuptools_scm (from the Python Packaging Authority). In order to determine the version setuptools_scm takes a look at three things:

• Latest tag (with a version number)
• The distance to this tag (e.g. number of revisions since latest tag)
• Workdir state (e.g. uncommitted changes since latest tag)

The above works optimally if you have a mechanism to automatically tag your releases, but you may choose to add the tags manually. In any case what you want is for setuptools_scm to pick up the latest tag (such as 2.1.12) and use it to update your library's version.

The example below illustrates what a typical set up would look like. I used semantic-delivery-gitlab (which uses semantic versioning based on commit messages) to tag the various commits, but other ways are possible. The master branch is treated as the release branch.

Configure setuptools_scm:

# pyproject.toml
[build-system]
requires = ["setuptools>=45", "wheel", "setuptools_scm>=6.2"]

[tool.setuptools_scm]
write_to = "my_library/__version__.py"

Get version:

# `my_library/__init__.py`
try:
    from my_library.__version__ import version as __version__
except ImportError:
    pass

Minimal .gitlab-ci.yaml:

# .gitlab-ci.yaml
stages:
  - build
  - release
  - publish

build:
  stage: build
  script:
    - pip install --upgrade pip
    - pip install setuptools setuptools_scm[toml] --upgrade
    - python setup.py bdist_wheel
  artifacts:
    expire_in: 7 days
    paths:
      - dist/*

.publish:
  stage: publish
  script:
    - WHEEL=$(ls dist)
    - publish_artifact.sh # Upload wheel to repository manager (e.g. artifactory)
 
publish-snapshot:
  <<: *publish
  except:
    - tags
    - master

publish-release:
  <<: *publish
  only:
    - tags

release:
  stage: release
  script:
    - npx @hutson/semantic-delivery-gitlab --token ${GITLAB_AUTH_TOKEN}
  only:
    - master
  when: manual # Manually trigger the tagging job for better control

You probably also want to add my_library/__version__.py to .gitignore. At the end of this process you may install the package and confirm that it has the right version with

>>> import my_library
>>> my_library.__version__
1.0.1

Answer 2

I like to use the bump2version package for this.

Here is my gitlab-ci.yml with the (almost) bare minimum setup:

image: python:latest

# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

# Pip's cache doesn't store the python packages
# https://pip.pypa.io/en/stable/reference/pip_install/#caching
#
# If you want to also cache the installed packages, you have to install
# them in a virtualenv and cache it as well.
cache:
  paths:
    - .cache/pip
    - venv/

before_script:
  - python --version
  - pip install virtualenv
  - virtualenv venv
  - source venv/bin/activate
  - pip install -r requirements.txt

stages:
  - build
  - release

upload package:
  stage: release
  script:
    - pip install twine bump2version
    - bump2version --tag release
    - python setup.py sdist bdist_wheel
    - TWINE_PASSWORD=${PYPI_TOKEN} TWINE_USERNAME=__token__ python -m twine upload --repository-url https://upload.pypi.org/legacy/ dist/*
    # Necessary to avoid a bug corrupting the version in setup.py. Just in case it's forgotten to do manually. Now we only have to explicitly bump version for major or minors.
    - bump2version patch
    - git config --global user.email "${GITLAB_USER_EMAIL}"
    - git config --global user.name "${GITLAB_USER_NAME}"
    - git remote set-url origin "https://gitlab-ci-token:${MY_PUSH_TOKEN}@gitlab.com/${CI_PROJECT_NAMESPACE}/${CI_PROJECT_NAME}.git"
    - git push -o ci.skip --tags origin HEAD:${CI_COMMIT_REF_NAME}
  artifacts:
    paths:
      - dist/*.whl
  only:
    - master

I also have a .bumpversion.cfg file in the root of my project with the following:

[bumpversion]
commit = True
tag = False
current_version = 0.1.0-dev0
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-(?P<release>[a-z]+)(?P<build>\d+))?
serialize = 
    {major}.{minor}.{patch}-{release}{build}
    {major}.{minor}.{patch}

[bumpversion:file:setup.py]

[bumpversion:part:build]

[bumpversion:part:release]
optional_value = gamma
values = 
    dev
    gamma

Two custom variables are used. They need to be added to the CI Variables in the repo settings. If you want to use them on a non-protected branch, make sure to uncheck the protected check.

• MY_PUSH_TOKEN - This is a personal access token created in my profile. It has the read_repository and write_repository permissions. I'm the owner/maintainer of this repository so hence it grants the permission to push on this repo.

• PYPI_TOKEN - Optional, needed to push the package to pypi.org.

Last but not least, worth mentioning:

• The example above uses a repo that is in a group, you may need to change the set-url origin address if you have a single repo not inside a group.

• The -o ci.skip argument prevents build pipeline triggers a loop

Usage:

• create feature branch
• push code
• create merge request
• merge MR to master

ci job takes care of packaging, releasing, uploading and and bumping to the next patch.

To bump major or minor, manually invoke it from command line locally in feature branch and push it.

The bump2version tool automatically takes care of the tagging too.

Some resources that I used to get to this solution:

• https://medium.com/@tishkov.pavel/autotagging-in-gitlab-step-by-step-guide-c3a8dd47087a - to get git in the ci working.
• https://williamhayes.medium.com/versioning-using-bumpversion-4d13c914e9b8 - to help me understand bump2version better