How Do I Query GitLab (issues, milestones, etc) from within a GitLab Pipeline

Question

Problem Summary

I am creating my first GitLab pipeline and setting it up to do a release. I would like to add a change log to the published release information. (found in Deployments -> Releases in the GitLab UI).

I would like to, from the pipeline, query the project for something like "closed issues since the date of the last release" and add those tickets to the published release.

Planned Approach

I am successfully using the "release-cli" tool from GitLab to create the release. I followed the example: "Create release metadata in custom script" on this page.

I found the description tag in the release element can be given a .md file, which seemed perfect for posting my change log. I have made a test .md file and added it as the release description and it looks great. It shows up right under "Evidence Collection" and so I planned to do "Change Log" as the header in my .md file. In the above example, GitLab is clearly using an .md file in their description element for extra information such as this.

Where I'm Stuck

I just can't figure out how to query the gitlab project from the pipeline (for issues or whatever).

I have been looking at GitLab's cli project and see likely has all the tools I need. I can not find any examples of using it in a pipeline. I cannot find for sure that its in a docker image that I can use similar to the release-cli tool.

I am stuck. The lack of information makes me wonder if I'm going down a dead end path. Is there anybody that has done something similar that can give me a basic example? Or an alternative?

Answer 1

To help others, I came up with a solution I'm happy with using gitlab's CLI tool. (forgive any .md formatting errors.. I really tried)

To make it work I had to:

• Create a docker image which included gitlab cli
• Add a couple of scripts to that image (genChangelog.sh, udpdateLabels.sh)
• Add it to the pipeline

I'm sure there are improvements to Dockerfile & scripts for more expert Docker/bash folks. I also tried to cut out as much stuff as possible that didn't speak to the core of the solution.

Following are snips from the various pieces I needed to get it to work. The result of all of this is

• Any issue that exists on the project with the 'release notes' label (and is closed) will get added to a changelog during every merge to development.
• The changelog effectively 'grows' during the development cycle and can be affected by adding issues to the project with appropriate title & 'release notes' label (and closing the issue). Also titles of existing issues tagged with 'release notes' can be updated to make them look nicer.
• When its time to run a release, the pipeline generates a final changelog, labeled with the release version. It then removes the 'release notes' label from all issues, and adds a new label with the version of the software that was just released. (We use the release tag from a maven:release command)
• The release cli docker image then takes that release changelog and adds it to the description field of the release 'widget' that lives in Deployments->Releases for that project in gitlab.

Dockerfile

FROM <base image with git and rpm available>

RUN rpm -ivh https://gitlab.com/gitlab-org/cli/-/releases/v1.25.0/downloads/glab_1.25.0_Linux_x86_64.rpm
COPY image/genChangelog.sh \
     image/updateLabels.sh \
     /usr/local/bin/
RUN mkdir -p /token

• token directory: Used by my pipeline to add an access token
• base image: I had a custom created image that I used. All it needs is git and rpm available. Also, my script uses bash shell.

Create Change log Script

• note: this depends on issues in the project being released having a label "release notes" attached.

'''

#!/bin/bash
# create temp files to put together the changelog.md
BUGS_FILE="$(mktemp)"
FEATURES_FILE="$(mktemp)"
CHANGELOG_FILE="$(mktemp)"

# read version string from command line or env variable
# this is simply to title the changelog
CHANGELOG_VERSION=${1:-$VERSION}

# login to gitlab with the access token written by the pipeline
glab auth login --hostname ${CI_SERVER_HOST} --stdin < /token/access_token.txt

# get all issues with 'release notes' label...
glab issue list -c --label "release notes" -F ids |\
while read id; do

    # write to a tmp file for parsing.  CLI returns strange formatting
    # also, there's no way to get the title directly in the cli (currently)
    glab issue view $id > issue.tmp

    # grep the title from the tmp file
    title=`grep ^title: issue.tmp | awk '{$1=$1};1' | sed -e "s/^title: //"`

    # also grep labels so we can see if this is a bug or story for nicer
    # release notes
    labels=`grep ^labels: issue.tmp`

    # write the id/title to either the bug, or features temp file
    # write in md format
    if [[ "$labels" =~ "type::Bug" ]]; then
       echo "- [#$id] $title" >> "$BUGS_FILE"
    elif [[ -n "$labels" ]]; then
       echo "- [#$id] $title" >> "$FEATURES_FILE"
    fi
done

# Generate the Changelog (really, just cat everything together)
anychange=false # if no labels with release notes, write something nice
echo "# Changes in Version: ${CHANGELOG_VERSION}" > "$CHANGELOG_FILE"

if [[ -s "$FEATURES_FILE" ]]; then
    echo "## New Functionality" >> "$CHANGELOG_FILE"
    # sort so in issue id order
    sort -t'#' -n -k2 "$FEATURES_FILE" >> "$CHANGELOG_FILE"
    anychange=true
fi
if [[ -s "$BUGS_FILE" ]]; then
    echo "## Bug Fixes" >> "$CHANGELOG_FILE"
    sort -t'#' -n -k2 "$BUGS_FILE" >> "$CHANGELOG_FILE"
    anychange=true
fi
if [[ $anychange == "false" ]]; then
    echo "- No notable changes since the previous version." >> "$CHANGELOG_FILE"
fi

'''

*note: this is run on every merge to development. It does not run in branches so we don't have to deal with merge conflicts. The changelog grows until we're ready to relase. When we release, the release pipeline will remove the 'release notes' label and replace it with the version that was just released.

Update Labels Script

(called only during release)

#!/usr/bin/bash

CI_SERVER_HOST=${CI_SERVER_HOST:-"<your gitlab host>"}

#either use environment variable or pass it in on command line
LABEL_VERSION=${1:-$VERSION}     

# login.  Using access token saved by pipeline
glab auth login --hostname ${CI_SERVER_HOST} --stdin < /token/access_token.txt

# Create new version label.  Don't fail if it exists
glab label create -c "#993300" \
     -d "Release notes for ${LABEL_VERSION} release." \
     -n "${LABEL_VERSION}" > /dev/null 2>&1 || true

## Add label to all tickets with a label "release notes" on them
## After adding, remove the "release notes" label
glab issue list -c --label "release notes" -F ids |\
while read id; do
  glab issue update $id --label "${LABEL_VERSION}" || true
  glab issue update $id --unlabel "release notes"
done

Pipeline (Merge to Development, Not a Release)

This reads the labels and creates and commits the changelog. Will also make the changelog available as an artifact that I can use in release notes on the gitlab releases page.

# Built in rules only enable this when on the default branch
update_changelog:
   image: ${DOCKER_REGISTRY}/gitlab-cli:latest
   stage: build
   needs:
      - job: get_version
      artifacts: true
   rules:
     - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
   variables:
      CHANGELOG_FILE: ".changelog.md"
   script:
       - echo "Creating changelog (${CHANGELOG_FILE}) for version=${VERSION}"
       - echo ${PIPELINE_ACCESS_TOKEN} > /token/access_token.txt
       - /usr/local/bin/genChangelog.sh ${VERSION} > ${CHANGELOG_FILE}
       - git config --global user.email "${CI_EMAIL}"
       - git config --global user.name "${CI_USERNAME}"
       - git remote set-url origin "https://${PIPELINE_ACCESS_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
       # use the || true in case there were no changes to the changelog
       - git add .changelog.md || true
       - git commit -m "Update Change Log" || true
       - git push origin HEAD:${CI_COMMIT_BRANCH} || true
   artifacts:
   paths:
   - ${CHANGELOG_FILE}

Pipeline (Release)

• Create the changelog again, making sure to label with release version.
  (same as above, different version passed in)
• Do your release (we use maven.. not interesting)
• Create the release in the 'Deployments->Releases' listing in gitlab for your project
• Update all the labels for the new release version

'''

# job depends on the release job to pass it the tag of the release
update_labels:
   image: ${DOCKER_REGISTRY}/gitlab-cli:latest
   stage: release
   needs:
      - job: perform_release
        artifacts: true
   script: 
     - echo ${PIPELINE_ACCESS_TOKEN} > /token/access_token.txt
     - echo "Updating labels after release ${TAG}"
     - /usr/local/bin/updateLabels.sh ${TAG}    

# This job is last.  It creates a tag in the repo as well as publishes release info to the deployments -> releases page
publish_release_info:
   stage: release
   image: registry.gitlab.com/gitlab-org/release-cli:latest
   needs:
     - job: perform_release  # to get the tag
       artifacts: true
     - job: update_changelog_release # to get the .changelog.md
       artifacts: true
   release:
      name: '${CI_PROJECT_NAME} v${RELEASE_VERSION}'
      description: .changelog.md
      tag_name: '$TAG'               # assumes tag exists from perform release step

'''