CODEWITHSUNDEEP
linuxbashshelljenkins
garci560
garci560

Reputation: 3213

Clean way of launching a shell script in background from Jenkins

What's the proper way to launch a script from jenkins, don't get the build hanging, and leave the process running? I can't seem to get it to work. Either the script doesn't run or the build hangs.

If I put in the build's "Execute shell" step bash relaunch.sh & or relaunch.sh > output.log & or nohup bash relaunch.sh &, nothing happens; build finishes, but the process doesn't run. I guess it can be related to Jenkins waiting for the error pipe to close.

If I do nohup bash relaunch.sh 2>&1 > output.log as suggested here, the output is properly redirected, but the build hangs (doesn't finish), and the process dies when I kill the build.

Adding export BUILD_ID=dontKillMe, as suggested here, here, and here, either to the "Execute shell" step or the script itself doesn't help either. The build hangs and the process dies when I kill the build. Needless to say, my knowledge of linux is very limited.

How do people do this in a clean way?

Upvotes: 29

Views: 33749

Answers (6)

yasin sula
yasin sula

Reputation: 11

I have implemented the solution as follows:

stage('Start AdminServer') {
  sh 'export JENKINS_NODE_COOKIE=dontKillMe;export BUILD_ID=dontKillMe;/app/jenkins/scripts/applyPatch/applyPatch.sh startAdmin' 
}

Upvotes: 1

msaba
msaba

Reputation: 1

your Jenkins command box should only contain ksh /path/to/script/script.sh

your script file should look like this:

#!/bin/bash
command > output.log &

Upvotes: 0

Miguel Ortiz
Miguel Ortiz

Reputation: 1482

I've solved a similar issue by executing:

mycommand -parameters &> /dev/null &

Somehow Jenkins was attached to the stdout of my script.

So I tried redirecting the standard output/error to /dev/null using &> /dev/null then adding the process to background using & and solved the problem, Jenkins ended the build without killing the process spawned.

PD: I tried adding an exit 0 at the end of the script, and it was executed, but that didn't free the Jenkins build, it kept attached to the subprocess output.

Upvotes: 0

GrueMaster
GrueMaster

Reputation: 101

Interesting solutions. Has anyone tried screen?

Here's how I run it (for RTL simulation using ModelSim-ASE):

screen -S $testname -d -m -L bash -c 'cd build_sim ; make; make sim'
echo Waiting for simulator ...

I then wait for our simulator to get to a waiting state by watching the screenlog.0

until ((`fgrep -c 'Ready for simulation' screenlog.0`)) ;do
    sleep 1
done

Then, when the other jobs that need the simulation are done running (or Jenkins detects a failure), in the post, run:

screen -S $testname -X kill

The only caveat is that if Jenkins somehow dies before the job cleans up, in theory you could have a simulator chewing up resources while waiting for something to happen.

To break it down a little: With screen, '-S' assigns a title to the session (or addresses a running session with that title), '-d' runs detached, '-m' spawn a fork first (which hides from Jenkins watchful eyes), and -L turns on logging (which is also handy in case something in the simulator breaks - you now have a log file artifact with the output). The rest is simply bash inside of the screen session.

Upvotes: 2

Inian
Inian

Reputation: 85895

A convenient way to achieve that is to change the environment variable BUILD_ID under Execute shell which Jenkins's ProcessTreeKiller is looking for.

By doing,

BUILD_ID=dontKillMe nohup bash relaunch.sh &

Jenkins will assume that the background job is not spawned by the build and will not kill them after finishing the job.

Thanks to Joshua for his observation that you could also use JENKINS_NODE_COOKIE as

JENKINS_NODE_COOKIE=dontKillMe

Upvotes: 36

user2671131
user2671131

Reputation: 145

I was having the exact same problem. I ended up fixing this by placing the following in the Jenkins execute shell box:

BUILD_ID=dontKillMe ./grid.sh

I moved the & inside the script file. Here's what the script looks like:

#!/bin/bash
java -jar selenium-server-standalone-3.0.1.jar -role hub &

Hopefully this helps someone!

Upvotes: 9

Related Questions