How do I configure a Jenkins multi-branch pipeline to build as a submodule?

Question

I have a project kuma organized with submodules:

• kuma
  • Jenkinsfile (configured to test kuma)
  • locales
  • kumascript
    • Jenkinsfile (configured to test kumascript)
  • A bunch of other files

I'd like to configure a multi-branch pipeline in Jenkins that watches for branches on the kumascript repo, to:

1. Check out the master branch of kuma
2. Update the locales to the commit in the master branch (a regular git submodule update --init)
3. Update the kumascript submodule to the branch to test
4. Run the Jenkinsfile in the kumascript branch

Is this possible? Is there a better way to do this?

Answer 1

Here's what worked for me.

First, the Jekinsfile is read from the commit, before checkout, so it is easy to use the one in the kumascript submodule, and much, much harder (impossible?) to read it from a different repo.

In Jenkins 2.68 with Git plugin 3.4.1, I setup a multibranch pipeline. The one source is Git, pointing to the kumascript repository:

"Discover branches" finds branches in the repository and starts builds for them.

"Wipe out repository and force clone" works around an issue where jgit doesn't fetch a submodule repository before checking it out, and thus the target commit isn't available. It causes an error that looks like this in the Jenkins logs:

> git fetch --no-tags --progress https://github.com/mdn/kumascript +refs/heads/*:refs/remotes/origin/*
Checking out Revision 998d9e539127805742634ef1c850221cf04ca2c7 (build-with-locales-1340342)
org.eclipse.jgit.errors.MissingObjectException: Missing unknown 998d9e539127805742634ef1c850221cf04ca2c7
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:158)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:227)
at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:859)
at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:772)

This issue appears to be reported in https://issues.jenkins-ci.org/browse/JENKINS-45729, and is reported fixed in Git client plugin 2.5.0.

Wiping out the repo appears to force the full fetch, and may be necessary when installing in a parent project.

Jenkins is now configured to create a build for each branch in the repository. To check it out as a submodule, the parent project will need to be manually checked out in the Jenkinsfile. I used Jenkin's "Pipeline Syntax" tool to help construct the command

After some formatting, this goes in my Jenkinsfile:

    stage("Prepare") {
        // Checkout Kuma project's master branch
        checkout([$class: 'GitSCM',
                  userRemoteConfigs: [[url: 'https://github.com/mozilla/kuma']],
                  branches: [[name: 'refs/heads/master']],
                  extensions: [[$class: 'SubmoduleOption',
                                disableSubmodules: false,
                                parentCredentials: false,
                                recursiveSubmodules: true,
                                reference: '',
                                trackingSubmodules: false]],
                                doGenerateSubmoduleConfigurations: false,
                  submoduleCfg: []
                 ])

        // Checkout KumaScript in subfolder
        dir('kumascript') {
          checkout scm
        }
    }

This checks out the kuma project and its submodules, and then uses the "vanilla" checkout to checkout the requested branch, but in the submodule directory:

From then on, if I want to run a command in the kuma repo, I run it:

    stage('Build') {
      sh 'make build-kumascript VERSION=latest'
    }

and if I want to run it in the kumascript submodule, I wrap it in dir:

    stage('Lint') {
      dir('kumascript') {
        sh 'make lint VERSION=latest'
        sh 'make lint-macros VERSION=latest'
      }
    }