How to run Vitest tests parallel in Gitlab with multiple jobs

Question

Our unit tests currently run in a single job on Gitlab. We are using React Vitest (800+ tests) but the job takes way to much time for our liking (7+ min). I saw that you can run multiple jobs in parallel in Gitlab. Anyone an idea on how to make a unit test parallel job in Gitlab that would devide our Vitest tests into smaller pieces and run them parallel?

Also a requisite is to eventually make 1 cobertura code coverage report that we then can use in Gitlab.

Answer 1

I found the solution myself. So first I added the following 2 job definitions in my gitlab-ci.yml file.

The first job will run the tests in parallel (with 7 jobs, this can be customizable). It also generates a report for every parallel test job and pushes it to the artifact. I found out that artifacts are shared between parallel jobs (and also every job in the same pipeline flow). I wanted to use cache instead but that didn't work out because it can not be shared as needed.

web:test:
  stage: test
  parallel: 7
  variables:
    VITE_CI_JOB_INDEX: $CI_NODE_INDEX
    VITE_CI_PARALLEL_SETTING: $CI_NODE_TOTAL
  script:
    - npm run test:coverage
  artifacts:
    expire_in: '1 hrs'
    paths:
      - $WEB_DIR/coverage/**
      - $WEB_DIR/test-report.xml

Second step is to merge all the code coverage reports into 1 main report and give it to Gitlab.

web:test:coverage:
  stage: test
  needs:
    - web:test
  script:
    - npm run test:coverage:merge
    - ../build/merge-code-coverage-reports.sh
  artifacts:
    expire_in: '5 days'
    paths:
      - $WEB_DIR/coverage/**
    reports:
      coverage_report:
        coverage_format: cobertura
        path: $WEB_DIR/coverage/cobertura-coverage.xml
  coverage: /All files[^|]*\|[^|]*\s+([\d\.]+)/

Now in the first job (web:test) we have a npm command: npm run test:coverage. This triggers vitest run --coverage and because it has the VITE_CI_JOB_INDEX and VITE_CI_PARALLEL_SETTING defined. It will use a custom sequencer in the Vitest config to slice the unit tests up in 7 chunks. One for eacht unit test parallel job in Gitlab.

const TESTS_PARALLEL_MODE = process.env.VITE_CI_JOB_INDEX && process.env.VITE_CI_PARALLEL_SETTING;

export default defineConfig({
  test: {
    globals: true,
    testTimeout: 2000,
    environment: 'jsdom',
    setupFiles: './tests/setup.ts',
    sequence: {
      shuffle: false,
      sequencer: TESTS_PARALLEL_MODE ? GitlabRunnerSequencer : null,
    },
    css: false,
    coverage: {
      all: true,
      reporter: TESTS_PARALLEL_MODE ? ['cobertura', 'json', 'text'] : ['lcov', 'text'],
      reportsDirectory: TESTS_PARALLEL_MODE ? `./coverage/${process.env.VITE_CI_JOB_INDEX}` : './coverage',
      provider: 'v8',
    },
  },
});

The sequencer.ts file looks like the following. It will just sort out the current chunk using the job index.

import { BaseSequencer, Vitest, WorkspaceSpec } from 'vitest/node';

class GitlabRunnerSequencer extends BaseSequencer {
  private gitlabJobIndex: number;

  private gitlabParallelSetting: number;

  constructor(ctx: Vitest) {
    super(ctx);
    this.gitlabJobIndex = Number(process.env.VITE_CI_JOB_INDEX || '0');
    this.gitlabParallelSetting = Number(process.env.VITE_CI_PARALLEL_SETTING || '1');
  }

  public async sort(files: WorkspaceSpec[]): Promise<WorkspaceSpec[]> {
    console.info(`VITE_CI_JOB_INDEX: ${this.gitlabJobIndex}`);
    console.info(`VITE_CI_PARALLEL_SETTING: ${this.gitlabParallelSetting}`);

    const sortedFiles = GitlabRunnerSequencer.sortByPath(files);
    const testChunk = this.getChunkForCurrentGitlabRunner(sortedFiles);

    console.info(`A total of ${testChunk.length} will be run`);

    return testChunk;
  }

  static sortByPath(tests: WorkspaceSpec[]): WorkspaceSpec[] {
    return tests.sort((a, b) => {
      if (a[0].path < b[0].path) {
        return -1;
      }
      if (a[0].path > b[0].path) {
        return 1;
      }
      return 0;
    });
  }

  getChunkForCurrentGitlabRunner(tests: WorkspaceSpec[]): WorkspaceSpec[] {
    const chunkSize = tests.length / this.gitlabParallelSetting;
    const currentChunkPositionRange = [Math.round(chunkSize * (this.gitlabJobIndex - 1)), Math.round(chunkSize * this.gitlabJobIndex)];

    console.info(`Running test chunk ${currentChunkPositionRange[0]} - ${currentChunkPositionRange[1]} for this runner`);

    return tests.filter((_, index) => index + 1 > currentChunkPositionRange[0] && index + 1 <= currentChunkPositionRange[1]);
  }
}

export default GitlabRunnerSequencer;

Then the final job is web:test:coverage which uses the test:coverage:merge which calls: node ./tests/merge-cobertura-reports.cjs:

const { createCoverageMap } = require('istanbul-lib-coverage');
const { createContext } = require('istanbul-lib-report');
const { create } = require('istanbul-reports');
const { resolve } = require('path');
const { sync } = require('glob');

console.log('Generating final report...');

const coverageMap = createCoverageMap();

const REPORTS_FOLDER = 'coverage';
const coverageDir = resolve(__dirname, `../${REPORTS_FOLDER}`);
const reportFiles = sync(`${coverageDir}/*/coverage-final.json`);

const normalizeReport = (report) => {
  const normalizedReport = { ...report };
  Object.entries(normalizedReport).forEach(([k, v]) => {
    if (v.data) normalizedReport[k] = v.data;
  });
  return normalizedReport;
};

reportFiles
  .map((reportFile) => {
    console.log(`Found report file: ${reportFile}`);
    return require(reportFile);
  })
  .map(normalizeReport)
  .forEach((report) => coverageMap.merge(report));
const context = createContext({
  coverageMap,
  dir: coverageDir,
});

create('cobertura', {}).execute(context);
console.log(`Cobertura coverage report generated and outputted to ${coverageDir}`);

Last thing is to echo the total coverage inside the job log, in order for Gitlab to use it in the analytics. This is done with the merge-code-coverage-reports.sh script. This could have also been done within the js file but I already made this in advance.

#!/bin/sh

REPORT_FILE="coverage/cobertura-coverage.xml"

echo "Retrieving line rate code coverage percentage..."

BRANCHE_RATE=$(grep -o 'branch-rate="[^"]*' $REPORT_FILE | sed 's/branch-rate="//')
LINE_COVERAGE=${BRANCHE_RATE:2:2}.${BRANCHE_RATE:4:6}

echo "All files          |   ${LINE_COVERAGE}"

echo "Cleaning up fractured reports..."

And now you can just change the number of parallel jobs to your liking.