CODEWITHSUNDEEP
bashjenkinsgroovyjenkins-pipeline
Stephen Nichols
Stephen Nichols

Reputation: 2483

Jenkins: Pipeline sh bad substitution error

A step in my pipeline uploads a .tar to an artifactory server. I am getting a Bad substitution error when passing in env.BUILD_NUMBER, but the same commands works when the number is hard coded. The script is written in groovy through jenkins and is running in the jenkins workspace.

sh 'curl -v --user user:password --data-binary ${buildDir}package${env.BUILD_NUMBER}.tar -X PUT "http://artifactory.mydomain.com/artifactory/release-packages/package${env.BUILD_NUMBER}.tar"'

returns the errors:

[Pipeline] sh
[Package_Deploy_Pipeline] Running shell script
/var/lib/jenkins/workspace/Package_Deploy_Pipeline@tmp/durable-4c8b7958/script.sh: 2: 
/var/lib/jenkins/workspace/Package_Deploy_Pipeline@tmp/durable-4c8b7958/script.sh: Bad substitution
[Pipeline] } //node
[Pipeline] Allocate node : End
[Pipeline] End of Pipeline
ERROR: script returned exit code 2

If hard code in a build number and swap out ${env.BUILD_NUMBER} I get no errors and the code runs successfully.

sh 'curl -v --user user:password --data-binary ${buildDir}package113.tar -X PUT "http://artifactory.mydomain.com/artifactory/release-packages/package113.tar"'

I use ${env.BUILD_NUMBER} within other sh commands within the same script and have no issues in any other places.

Upvotes: 80

Views: 138996

Answers (11)

ADV-IT
ADV-IT

Reputation: 831

This caused the error Bad Substitution:

pipeline {
     agent any

     environment {
        DOCKER_IMAGENAME = "mynginx:latest"
        DOCKER_FILE_PATH = "./docker"
     }
  
     stages {
         stage('DockerImage-Build') {
            steps {
             sh 'docker build -t ${env.DOCKER_IMAGENAME} ${env.DOCKER_FILE_PATH}'
            }
         }             
     }
 }

This fixed it: replace ' with " on sh command

    pipeline {
     agent any

     environment {
        DOCKER_IMAGENAME = "mynginx:latest"
        DOCKER_FILE_PATH = "./docker"
     }
  
     stages {
         stage('DockerImage-Build') {
            steps {
             sh "docker build -t ${env.DOCKER_IMAGENAME} ${env.DOCKER_FILE_PATH}"
            }
         }             
     }
 }

Upvotes: 2

Durgaprasad A
Durgaprasad A

Reputation: 1

I got similar issue. But my usecase is little different

            steps{
                sh '''#!/bin/bash -xe
                      VAR=TRIAL
                      echo $VAR
                      if [ -d /var/lib/jenkins/.m2/'\${params.application_name}' ]
                      then
                        echo 'working'
                        echo ${VAR}
                      else
                        echo 'not working'
                      fi
                      
                      
            '''
            }
        }

here I'm trying to declare a variable inside the script and also use a parameter from outside

After trying multiple ways

The following script worked

stage('cleaning com/avizva directory'){
            steps{
                sh """#!/bin/bash -xe
                      VAR=TRIAL
                      echo \$VAR
                      if [ -d /var/lib/jenkins/.m2/${params.application_name} ]
                      then
                        echo 'working'
                        echo \${VAR}
                      else
                        echo 'not working'
                      fi
                      
                      
            """
            }
        }

changes made :

Replaced triple single quotes --> triple double quotes

Whenever I want to refer to local variable I used escape character $VAR --> \$VAR

Upvotes: 0

StNickolay
StNickolay

Reputation: 960

suggestion from @avivamg didn't worked for me, here is the syntax which works for me:

 sh "python3 ${env.WORKSPACE}/package.py --product productname " +
    "--build_dir ${release_build_dir} " +
    "--signed_product_dir ${signed_product_dir} " +
    "--version ${build_version}"

Upvotes: 0

Jason Mullings
Jason Mullings

Reputation: 1004

The Jenkins Script is failing inside the "sh" command-line E.g:

sh 'npm run build' <-- Fails referring to package.json

Needs to be changed to:

sh 'npm run ng build....'

... ng $PATH is not found by the package.json.

Upvotes: -1

Promise Preston
Promise Preston

Reputation: 28870

I had this same issue when working on a Jenkins Pipeline for Amazon S3 Application upload.

My script was like this:

pipeline {  
  agent any
  parameters {
    string(name: 'Bucket', defaultValue: 's3-pipeline-test', description: 'The name of the Amazon S3 Bucket')
    string(name: 'Prefix', defaultValue: 'my-website', description: 'Application directory in the Amazon S3 Bucket')
    string(name: 'Build', defaultValue: 'public/', description: 'Build directory for the application')
  }  
  stages {  
    stage('Build') {  
      steps {
        echo 'Running build phase'
        sh 'npm install' // Install packages
        sh 'npm run build' // Build project 
        sh 'ls' // List project files
      }  
    }  
    stage('Deploy') {  
      steps {  
        echo 'Running deploy phase'
        withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', accessKeyVariable: 'AWS_ACCESS_KEY_ID', credentialsId: 'AWSCredentials', secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) { 
          sh 'aws s3 ls' // List AWS S3 buckets
          sh 'aws s3 sync "${params.Build}" s3://"${params.Bucket}/${params.Prefix}" --delete' // Sync project files with AWS S3 Bucket project path
        }  
      }   
    }
  }
  post { 
    success {  
      echo 'Deployment to Amazon S3 suceeded'  
    }  
    failure {  
      echo 'Deployment to Amazon S3 failed'  
    }  
  }  
}

Here's how I fixed it:

Seeing that it's an interpolation call of variables, I had to substitute the single quotation marks (' ') in this line of the script:

sh 'aws s3 sync "${params.Build}" s3://"${params.Bucket}/${params.Prefix}" --delete' // Sync project files with AWS S3 Bucket project path

to double quotation marks (" "):

sh "aws s3 sync ${params.Build} s3://${params.Bucket}/${params.Prefix} --delete" // Sync project files with AWS S3 Bucket project path

So my script looked like this afterwards:

pipeline {  
  agent any
  parameters {
    string(name: 'Bucket', defaultValue: 's3-pipeline-test', description: 'The name of the Amazon S3 Bucket')
    string(name: 'Prefix', defaultValue: 'my-website', description: 'Application directory in the Amazon S3 Bucket')
    string(name: 'Build', defaultValue: 'public/', description: 'Build directory for the application')
  }  
  stages {  
    stage('Build') {  
      steps {
        echo 'Running build phase'
        sh 'npm install' // Install packages
        sh 'npm run build' // Build project 
        sh 'ls' // List project files
      }  
    }  
    stage('Deploy') {  
      steps {  
        echo 'Running deploy phase'
        withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', accessKeyVariable: 'AWS_ACCESS_KEY_ID', credentialsId: 'AWSCredentials', secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) { 
          sh 'aws s3 ls' // List AWS S3 buckets
          sh "aws s3 sync ${params.Build} s3://${params.Bucket}/${params.Prefix} --delete" // Sync project files with AWS S3 Bucket project path
        }  
      }   
    }
  }
  post { 
    success {  
      echo 'Deployment to Amazon S3 suceeded'  
    }  
    failure {  
      echo 'Deployment to Amazon S3 failed'  
    }  
  }  
}

That's all

I hope this helps

Upvotes: 5

Aviv
Aviv

Reputation: 14477

In order to Pass groovy parameters into bash scripts in Jenkins pipelines (causing sometimes bad substitions) You got 2 options:

The triple double quotes way [ " " " ] OR the triple single quotes way [ ' ' ' ]

  1. In triple double quotes you can render the normal parameter from groovy using ${someVariable} ,if it's environment variable ${env.someVariable} , if it's parameters injected into your job ${params.someVariable}

example:

     def YOUR_APPLICATION_PATH= "${WORKSPACE}/myApp/"

      sh """#!/bin/bash
      cd ${YOUR_APPLICATION_PATH}
      npm install
      """
  1. In triple single quotes things getting little bit tricky, you can pass the parameter to environment parameter and using it by "\${someVaraiable}" or concating the groovy parameter using ''' + someVaraiable + '''

examples:

   def YOUR_APPLICATION_PATH= "${WORKSPACE}/myApp/"

   sh '''#!/bin/bash
          cd ''' + YOUR_APPLICATION_PATH + '''
          npm install
    '''

OR

   pipeline{
     agent { node { label "test" } }
     environment {
       YOUR_APPLICATION_PATH = "${WORKSPACE}/myapp/"
     }

     continue...
     continue...
     continue...

     sh '''#!/bin/bash
          cd "\${YOUR_APPLICATION_PATH}"
          npm install
    '''

    //OR
    sh '''#!/bin/bash
          cd "\${env.YOUR_APPLICATION_PATH}"
          npm install
    '''

Upvotes: 56

Aamir M Meman
Aamir M Meman

Reputation: 1831

I was having the issue with showing the {env.MAJOR_VERSION} in an artifactory of jar file . show I approaches by keeping of environment step in Jenkinsfile.

pipeline {
agent any
environment {
MAJOR_VERSION = 1
}

stages {
stage('build') {
  steps {
      sh 'ant -f build.xml -v'
 }
}
}
post {
 always{
   archiveArtifacts artifacts: 'dist/*.jar', fingerprint: true
 }
}
}

I got the issue solved and then it was not showing me bad substitution in my Jenkins build output. so environment step plays a more role in Jenkinsfile.

Upvotes: 2

Gary Gan
Gary Gan

Reputation: 131

I can definitely tell you, it's all about sh shell and bash shell. I fixed this problem by specifying #!/bin/bash -xe as follows:

node {
    stage("Preparing"){
        sh'''#!/bin/bash -xe
            colls=( col1 col2 col3 )
            for eachCol in ${colls[@]}
            do
              echo $eachCol
            done
        '''
    }      
}

Upvotes: 13

kenorb
kenorb

Reputation: 166379

Usually the most common issue for:

Bad substitution

error is to use sh instead of bash.

Especially when using Jenkins, if you're using Execute shell, make sure your Command starts with shebang, e.g. #!/bin/bash -xe or #!/usr/bin/env bash.

Upvotes: 23

patric.schenke
patric.schenke

Reputation: 972

Actually, you seem to have misunderstood the env variable. In your sh block, you should access ${BUILD_NUMBER} directly.

Reason/Explanation: env represents the environment inside the script. This environment is used/available directly to anything that is executed, e.g. shell scripts.

Please also pay attention to not write anything to env.*, but use withEnv{} blocks instead.

Upvotes: 28

Stephen Nichols
Stephen Nichols

Reputation: 2483

This turned out to be a syntax issue. Wrapping the command in ''s caused ${env.BUILD_NUMBER to be passed instead of its value. I wrapped the whole command in "s and escaped the nested. Works fine now.

sh "curl -v --user user:password --data-binary ${buildDir}package${env.BUILD_NUMBER}.tar -X PUT \"http://artifactory.mydomain.com/artifactory/release-packages/package${env.BUILD_NUMBER}.tar\""

Upvotes: 123

Related Questions