CODEWITHSUNDEEP
jenkinsgroovyshared-librariesjenkins-pipeline
schnatterer
schnatterer

Reputation: 7859

Load Jenkins Pipeline Shared Library from same repository

TL;DR Is there a way to import code into the Jenkinsfile from the local repository (other than the load step)?

Why?

I've experienced that for complex builds the Jenkinsfile gets kind of bulky and not very maintainable.
Now that the build job is code, it would be wonderful to have the same means as for other code. That is, I would like to divide it into smaller (more maintainable) units and unit test them.

What I tried

My Questions

Example directory structure

Similar to the directory structure described for shared libs I would like to have the following in a single repository.

(root)
+- someModule
|   +- ...
+- jenkins           # Classes/Scripts used by Jenkins in a separate module
|   +- src                       # Groovy source files
|      +- org
|          +- foo
|              +- Bar.groovy     # for org.foo.Bar class
|   +- test                      # Groovy test files
|      +- org
|          +- foo
|              +- BarTest.groovy # Test for org.foo.Bar class
|   +- pom.xml or build.groovy   # Build for local library
+- Jenkinsfile     # Build "someModule", uses classes from "jenkins" module

Upvotes: 59

Views: 44537

Answers (9)

Velan
Velan

Reputation: 21

Wrote a small plugin(a custom retriever) to solve this issue. Need to paste the library files inside jenkins agent file system and mention the library path while loading the library in pipeline script like below.

For testing create logger.groovy under C:\test-library\vars\ directory with below content

#!/groovy
def info(message) {
    echo message
}

Jenkins pipeline script:

pipeline {
    agent any
    stages {
        stage("Test FileSystemRetriever"){
            steps{
                script{
                    library identifier: '[email protected]', retriever: fileSystemRetriever(
                    libraryPath: 'C:\\test-library')
                    log.info("info message")
                }
            }
        }
    }
}

Github repository for this plugin can be found here - https://github.com/Velan987/FileSystemRetriever

Upvotes: 0

Max Cascone
Max Cascone

Reputation: 833

My shared library pipeline can test itself. It's a bit clunky but it works.

All I do is import the library itself into the Jenkinsfile in the library's repo. The actual pipeline is defined in /vars as myPipeline.groovy - it's in Declarative syntax, uses the Delegate pattern provided by Jenkins to ingest parameters, and every effort is made to put all actual code into library steps, leaving the pipeline DSL as clean as possible.

// repo's Jenkinsfile
library 'my-shared-library'

myPipeline {
  myParam1 = 'value1'
  myParam2 = 'value2'
}

It's important to remember to pull the branch you're working on when you're testing something new:

// repo's Jenkinsfile
library 'my-shared-library@my-new-branch`
...

You can write actual unit tests into the /test/com/myOrg/ dir; I use the plain JenkinsPipelineUnit framework, it gets the job done, but it's clunky. I landed on using Gradle to perform the build and tests. It's slow, but gets the job done.

Or, you can test your /vars steps in the actual Jenkinsfile in an ad-hoc pipeline. This is sometimes easier than dealing with mocking everything out in units, especially if you end up mocking what you want to be testing, which defeats the purpose.

load() should definitely be providing the loaded scripts for editing in Replay. If not, something else is going on.

Lastly, I highly recommend setting up a Jenkins instance on your dev machine, so you can test changes without pushing them. Especially if you create an ad-hoc pipe to test your steps, you can then edit the pipeline in Replay, creating the shortest feedback loop for yourself. To get even faster, get familiar with testing groovy in the Script Console, or using Gradle run to actually run more complex code from the command line. Anything to avoid waiting for code to push and pipelines to run will make your development go faster.

note of warning on the script console, readJson/readYaml don't work right, and I'm sure others too. When i need to test stuff like that, i tend to use ad-hoc pipes to quickly iterate on the code.

Upvotes: 0

Igwe Kalu
Igwe Kalu

Reputation: 14868

Is there a way to import code into the Jenkinsfile from the local repository (other than the load step)?

Is there a way (or workaround) to create a shared library in the same repository as the Jenkinsfile and import this library into the Jenkinsfile?

Yes. Provided that the "directory structure of a Shared Library repository" is observed according to specification, then it's absolutely feasible and no workaround is required to get it working. Basically, your directory structure would require an adjustment along the following lines:

+- src                     # Groovy source files
|   +- org
|       +- foo
|           +- Bar.groovy  # for org.foo.Bar class
+- vars
|   +- foo.groovy          # for global 'foo' variable
|   +- foo.txt             # help for 'foo' variable
+- resources               # resource files (external libraries only)
|   +- org
|       +- foo
|           +- bar.json    # static helper data for org.foo.Bar
+- someModule
|   +- ...
|
|
+- Jenkinsfile

This answer is not based on a conjecture. Although it's not documented, I have applied this pattern on multiple projects and training, and it works. You do not need to tinker with your job configuration in anyway than is usual.

Upvotes: 0

Vaclav Bartacek
Vaclav Bartacek

Reputation: 121

I've successfully tested a simple workaround for GIT repos having both protocols - HTTPS and SSH (I'm using BitBucket) - just configure the Jenkins job as follows (pointing to the same repository but forcing different fetch methods):

  • In the Branch Sources add "Checkout over SSH" option

  • In the Pipeline Libraries -> Source Code Management -> Project Repository use HTTPS protocol - e.g. something like https://...

Upvotes: 0

lqbweb
lqbweb

Reputation: 1708

I found the version from Pawel has problems if you are checking out the pipeline from SCM (not embedded in Jenkins job UI config). This is my version for those cases:

node {
    def scriptFolder = sh(script: "echo \$(pwd | sed 's,\\(.*\\)\\(@[0-9]*\$\\),\\1,')@script/\$(sed -n '2,\$p' ../config.xml | xmllint --xpath '(//scriptPath/text())' - | xargs dirname)", returnStdout: true).trim()
    sh("cd ${scriptFolder} && ls -l vars/ && if [ -d .git ]; then rm -rf .git; fi; git init && git add --all . && git commit -m init &> /dev/null")
    library identifier: 'local-lib@master', retriever: modernSCM([$class: 'GitSCMSource', remote: scriptFolder])

    stage('Build') {
        log.info("called shared lib") // use the loaded library
    }
}

For these cases, the pipeline itself is checkout out in a different workspace (same directory name but with @script in the name) than what the pipeline itself defines.

Upvotes: 0

Jesse Glick
Jesse Glick

Reputation: 25461

Workaround:

library identifier: 'shared-library@version', retriever: legacySCM(scm)

The approach currently taken in PR 37 will not work properly with build agents, and anyway will only work for scripts using the library step, not the @Library annotation.

By the way files loaded from the load step do appear in Replay. But it is true that your script cannot statically refer to types defined in such files. In other words, you could simulate library vars/*.groovy but not src/**/*.groovy—the same limitation as the current PR 37.

Upvotes: 13

Karol Gil
Karol Gil

Reputation: 95

You may take a look at plugin I wrote, that enables using subdirectories of repo where your pipeline is as shared libraries: https://github.com/karolgil/SharedLibrary

After building and installing it, you can simply put following in your pipeline:

@SharedLibrary('dir/in/repo') _

To start using dir/in/repo as shared library for your pipelines.

Upvotes: 5

Pawel Wiejacha
Pawel Wiejacha

Reputation: 722

I guess that proper way to do that is to implement a custom SCMRetriever.

However, you can use the following hack:

Assuming jenkins/vars/log.groovy in your local repo contains:

def info(message) {
    echo "INFO: ${message}"
}

Your Jenkinsfile can load that shared library from the jenkins/ directory using library step:

node('node1') { // load library
    checkout scm
    // create new git repo inside jenkins subdirectory
    sh('cd jenkins && git init && git add --all . && git commit -m init &> /dev/null') 
    def repoPath = sh(returnStdout: true, script: 'pwd').trim() + "/jenkins"
    library identifier: 'local-lib@master', retriever: modernSCM([$class: 'GitSCMSource', remote: repoPath])
}

node('node2') {
    stage('Build') {
        log.info("called shared lib") // use the loaded library
    }
}

Upvotes: 9

iocanel
iocanel

Reputation: 580

Wanted to do the same and ended up creating this:

https://github.com/jenkinsci/workflow-cps-global-lib-plugin/pull/37

and here is how I use it:

https://github.com/syndesisio/syndesis-pipeline-library/blob/master/Jenkinsfile#L3

In my case I wanted to create a Jenkinsfile that actually tests the pipeline library that the repository contains.

Let me know what you think and feel free to add your comments on the PR too.

Upvotes: 2

Related Questions