CODEWITHSUNDEEP
jenkins
blokkie
blokkie

Reputation: 5565

How to stop an unstoppable zombie job on Jenkins without restarting the server?

Our Jenkins server has a job that has been running for three days, but is not doing anything. Clicking the little X in the corner does nothing, and the console output log doesn't show anything either. I've checked on our build servers and the job doesn't actually seem to be running at all.

Is there a way to tell jenkins that the job is "done", by editing some file or lock or something? Since we have a lot of jobs we don't really want to restart the server.

Upvotes: 265

Views: 297599

Answers (30)

Sarvesh Mishra
Sarvesh Mishra

Reputation: 81

I have created a script that runs hourly to delete zombie jobs because they don't show on the UI homepage. I have tried using the doStop() function, but it only sends an abort command that does not stop the pipeline from running.

import jenkins.model.Jenkins
import hudson.model.*

def abortLongRunningJobs() {
    def currentTime = System.currentTimeMillis()
    def threshold = 3 * 60 * 60 * 1000 // 3 hours in milliseconds


    Jenkins.instance.getAllItems(Job).each { job ->
        def runningBuilds = job.getBuilds().findAll { build ->
            (build.isBuilding()) && (currentTime - build.getTimeInMillis() > threshold)
        }
        runningBuilds.each { build ->
                
            Jenkins.instance.getItemByFullName(job.fullName)
                .getBuildByNumber(build.number)
                .finish(
                        hudson.model.Result.ABORTED,
                        new java.io.IOException("Aborting long running build")
                );
            println("Forcefully aborted job: ${job.fullName} - Build #${build.number}")
        }
    }
}

abortLongRunningJobs()

Upvotes: 2

Takman
Takman

Reputation: 1076

Go to Manage Jenkins then Script Console

Jenkins.instance.getItemByFullName(FULL_JOB_NAME)
    .getBuildByNumber(BUILD_NUMBER)
    .delete()

You can find FULL_JOB_NAME on your job page, right after Full project name: text

PS: .finish() method isn't recognized.

Upvotes: 2

Alexandru Bantiuc
Alexandru Bantiuc

Reputation: 3987

I had also the same problem and fix it via Jenkins Console.

Go to "Manage Jenkins" > "Script Console" and run a script:

Jenkins.instance.getItemByFullName("JobName")
        .getBuildByNumber(JobNumber)
        .finish(hudson.model.Result.ABORTED, 
                new java.io.IOException("Aborting build")
);

You'll have just specify your JobName and JobNumber.

Upvotes: 381

Frankenstein
Frankenstein

Reputation: 693

Three step program

First, abort the build via script console:

Jenkins.instance.getItemByFullName("JobName")
            .getBuildByNumber(JobNumber)
            .finish(
                    hudson.model.Result.ABORTED,
                    new java.io.IOException("Aborting build")
            );

Second, delete the build via script console:

Jenkins.instance.getItemByFullName(jobname).getBuildByNumber(buildnum).delete();

Make sure the build is not reachable in the ui.

Third, restart jenkins, for example via the ui: [jenkins_url]/restart

Upvotes: 1

Nikhil Singh
Nikhil Singh

Reputation: 51

If the "X" button is not working and the job is stuck, then just delete the specific build number. It will free up the executor.

In my case, even though the job was completed, it was still stuck in the executor for hours. Deleting the build worked for me.

Upvotes: 0

Dibakar Aditya
Dibakar Aditya

Reputation: 4203

Without having to use the script console or additional plugins, you can simply abort a build by entering /stop, /term, or /kill after the build URL in your browser.

Quoting verbatim from the above link:

Pipeline jobs can by stopped by sending an HTTP POST request to URL endpoints of a build.

  • <BUILD ID URL>/stop - aborts a Pipeline.
  • <BUILD ID URL>/term - forcibly terminates a build (should only be used if stop does not work.
  • <BUILD ID URL>/kill - hard kill a pipeline. This is the most destructive way to stop a pipeline and should only be used as a last resort.

Upvotes: 77

Aviel Yosef
Aviel Yosef

Reputation: 581

This works for me everytime:

Thread.getAllStackTraces().keySet().each() {
    if (it.name.contains('YOUR JOBNAME')) {  
      println "Stopping $it.name"
      it.stop()
    }
}

Thanks to funql.org

Upvotes: 5

Marc Lalibert&#233;
Marc Lalibert&#233;

Reputation: 113

None of these solutions worked for me. I had to reboot the machine the server was installed on. The unkillable job is now gone.

Upvotes: 0

Poulad
Poulad

Reputation: 1311

Using the Script console at https://my-jenkins/script

import hudson.model.Job
import org.jenkinsci.plugins.workflow.job.WorkflowRun

Collection<Job> jobs = Jenkins.instance.getItem('My-Folder').getAllJobs()
for (int i = 0; i < jobs.size(); i++) {
  def job = jobs[i]
  for (int j = 0; j < job.builds.size(); j++) {
    WorkflowRun build = job.builds[j]
    if (build.isBuilding()) {
      println("Stopping $job ${build.number}")
      build.setResult(Result.FAILURE)
    }
  }
}

Upvotes: 1

Sergey Pleshakov
Sergey Pleshakov

Reputation: 8948

VERY SIMPLE SOLUTION

The reason I was seeing this issue was incorrect http link on the page instead of https that should stop the job. All you need to do is to edit onclick attribute in html page, by following

  1. Open up a console log of the job (pipeline) that got hang
  2. Click whatever is available to kill the job (x icon, "Click here to forcibly terminate running steps" etc) to get "Click here to forcibly kill entire build" link visible (it's NOT gonna be clickable at the moment)
  3. Open the browser's console (use any one of three for chrome: F12; ctrl + shift + i; menu->more tools->developer tools)
  4. Locate "Click here to forcibly kill entire build" link manually or using "select an element in the page" button of the console
  5. Double click on onclick attribute to edit its value
  6. Append s to http to have https
  7. Press enter to submit the changes
  8. Click "Click here to forcibly kill entire build" link

Use screenshot for reference enter image description here

Upvotes: 3

Markus
Markus

Reputation: 4661

In case you got a Multibranch Pipeline-job (and you are a Jenkins-admin), use in the Jenkins Script Console this script:

Jenkins.instance
.getItemByFullName("<JOB NAME>")
.getBranch("<BRANCH NAME>")
.getBuildByNumber(<BUILD NUMBER>)
.finish(hudson.model.Result.ABORTED, new java.io.IOException("Aborting build"));

From https://issues.jenkins-ci.org/browse/JENKINS-43020

If you aren't sure what the full name (path) of the job is, you may use the following snippet to list the full name of all items:

  Jenkins.instance.getAllItems(AbstractItem.class).each {
    println(it.fullName)
  };

From https://support.cloudbees.com/hc/en-us/articles/226941767-Groovy-to-list-all-jobs

Upvotes: 65

de-jcup
de-jcup

Reputation: 1865

I had same issue at the last half hour...

Was not able to delete a zombie build running in my multi-branch pipeline. Even Server restarts by UI or even from commandline via sudo service jenkins restart did block the execution... The build was not stoppable... It always reapeared.

Used Version: Jenkins ver 2.150.2

I was very annoyed, but... when looking into the log of the build I found something intersting at the end of the log:

Logfile output of an zombie build and showing restart did not stop it

The red marked parts are the "frustrating parts"... As you can see I always wanted to Abort the build from UI but it did not work...

But there is a hyperlink with text Click here to forcibly terminate running steps...(first green one) Now I pressed the link...) After the link execution a message about Still paused appeared with another Link Click here to forcibily kill entire build (second green one) After pressing this link also the build finally was hard killed...

So this seems to work without any special plugins (except the multibranch-pipeline build plugin itself).

Upvotes: 3

t0r0X
t0r0X

Reputation: 4752

Recently I came across a node/agent which had one executor occupied for days by a build "X" of a pipeline job, although that jobs page claimed build "X" did not exist anymore (discarded after 10 subsequent builds (!), as configured in the pipeline job). Verified that on disk: build "X" was really gone.

The solution: it was the agent/node which wrongly reported that the occupied executor was busy running build "X". Interrupting that executor's thread has immediately released it.

def executor = Jenkins.instance.getNode('NODENAME').computer.executors.find {
    it.isBusy() && it.name.contains('JOBNAME')
}

println executor?.name
if (executor?.isBusy()) executor.interrupt()

Other answers considered:

  • The answer from @cheffe: did not work (see next point, and update below).
  • The answers with Thread.getAllStackTraces(): no matching thread.
  • The answer from @levente-holló and all answers with getBuildByNumber(): did not apply as the build wasn't really there anymore!
  • The answer from @austinfromboston: that came close to my needs, but it would also have nuked any other builds running at the moment.

Update:
I experienced again a similar situation, where a Executor was occupied for days by a (still existing) finished pipeline build. This code snippet was the only working solution.

Upvotes: 5

St&#233;phane
St&#233;phane

Reputation: 2108

I had many zombi-jobs, so I used the following script:

for(int x = 1000; x < 1813; x = x + 1) {
    Jenkins .instance.getItemByFullName("JOBNAME/BRANCH")
    .getBuildByNumber(x)
    .finish(hudson.model.Result.ABORTED, new java.io.IOException("Aborting build"))
}

Upvotes: 1

user3360767
user3360767

Reputation: 986

Enter the blue-ocean UI. Try to stop the job from there.

Upvotes: -4

Zahra
Zahra

Reputation: 7197

Go to "Manage Jenkins" > "Script Console" to run a script on your server to interrupt the hanging thread.

You can get all the live threads with Thread.getAllStackTraces() and interrupt the one that's hanging.

Thread.getAllStackTraces().keySet().each() {
  t -> if (t.getName()=="YOUR THREAD NAME" ) {   t.interrupt();  }
}

UPDATE:

The above solution using threads may not work on more recent Jenkins versions. To interrupt frozen pipelines refer to this solution (by alexandru-bantiuc) instead and run:

Jenkins.instance.getItemByFullName("JobName")
                .getBuildByNumber(JobNumber)
                .finish(
                        hudson.model.Result.ABORTED,
                        new java.io.IOException("Aborting build")
                );

Upvotes: 291

Kenneth King
Kenneth King

Reputation: 126

Had this same issue but there was not stack thread. We deleted the job by using this snippet in the Jenkins Console. Replace jobname and buil dnumber with yours.

def jobname = "Main/FolderName/BuildDefinition"
def buildnum = 6
Jenkins.instance.getItemByFullName(jobname).getBuildByNumber(buildnum).delete();

Upvotes: 4

Tom Bates
Tom Bates

Reputation: 576

Here is how I fixed this issue in version 2.100 with Blue Ocean

  • The only plugins I have installed are for bitbucket.
  • I only have a single node.

ssh into my Jenkins box
cd ~/.jenkins (where I keep jenkins)
cd job/<job_name>/branches/<problem_branch_name>/builds
rm -rf <build_number>

After this, you can optionally change the number in nextBuildNumber (I did this)

Finally, I restarted jenkins (brew services restart jenkins) This step will obviously be different depending how you manage and install Jenkins.

Upvotes: 0

jayhendren
jayhendren

Reputation: 4511

The top answer almost worked for me, but I had one major problem: I had a very large number (~100) of zombie jobs due to a particularly poorly-timed Jenkins restart, so manually finding the job name and build number of each and every zombie job and then manually killing them was infeasible. Here's how I automatically found and killed the zombie jobs:

Jenkins.instance.getItemByFullName(multibranchPipelineProjectName).getItems().each { repository->
  repository.getItems().each { branch->
    branch.builds.each { build->
      if (build.getResult().equals(null)) {
        build.doKill()
      }
    }
  }
}

This script loops over all builds of all jobs and uses getResult().equals(null) to determine whether or not the job has finished. A build that's in the queue but not yet started will not be iterated over (since that build won't be in job.builds), and a build that's finished already will return something other than null for build.getResult(). A legitimately running job will also have a build result of null, so make sure you have no running jobs that you don't want to kill before running this.

The multiple nested loops are mainly necessary to discover every branch/PR for every repository in a Multibranch Pipeline project; if you're not using Multibranch Pipelines you can just loop over all your jobs directly with something like Jenkins.instance.getItems().each.

Upvotes: 8

austinfromboston
austinfromboston

Reputation: 3780

Alexandru Bantiuc's answer worked well for me to stop the build, but my executors were still showing up as busy. I was able clear the busy executor status using the following

server_name_pattern = /your-servers-[1-5]/
jenkins.model.Jenkins.instance.getComputers().each { computer ->
  if (computer.getName().find(server_name_pattern)) {
    println computer.getName()
    execList = computer.getExecutors()      
    for( exec in execList ) {
      busyState = exec.isBusy() ? ' busy' : ' idle'
      println '--' + exec.getDisplayName() + busyState
      if (exec.isBusy()) {
        exec.interrupt()
      }
    }
  }
}

Upvotes: 5

funql.org
funql.org

Reputation: 438

The first proposed solution is pretty close. If you use stop() instead of interrupt() it even kills runaway threads, that run endlessly in a groovy system script. This will kill any build, that runs for a job. Here is the code:

Thread.getAllStackTraces().keySet().each() {
    if (it.name.contains('YOUR JOBNAME')) {  
      println "Stopping $it.name"
      it.stop()
    }
}

Upvotes: 36

Levente Holl&#243;
Levente Holl&#243;

Reputation: 724

If you have an unstoppable Pipeline job, try the following:

  1. Abort the job by clicking the red X next to the build progress bar
  2. Click on "Pause/resume" on the build to pause
  3. Click on "Pause/resume" again to resume the build

Pause/Resume pipeline job

Jenkins will realize that the job should be terminated and stops the build

Upvotes: 18

mugi
mugi

Reputation: 474

Once I encounterred a build which could not be stopped by the "Script Console". Finally I solved the problem with these steps:

ssh onto the jenkins server
cd to .jenkins/jobs/<job-name>/builds/
rm -rf <build-number>
restart jenkins

Upvotes: 32

Krzysztof Jabłoński
Krzysztof Jabłoński

Reputation: 1941

I usually use jenkins-cli in such cases. You can download the jar from a page http://your-jenkins-host:PORT/cli . Then run

java -jar jenkins-cli.jar delete-builds name_of_job_to_delete hanging_job_number

Auxiliary info:

You may also pass a range of builds like 350:400. General help available by running

java -jar jenkins-cli.jar help

Context command help for delete-builds by

java -jar jenkins-cli.jar delete-builds

Upvotes: 3

GUOHUI QIAN
GUOHUI QIAN

Reputation: 1

You can just copy the job and delete the old one. If it doesn't matter that you lost the old build logs.

Upvotes: -1

Jesse Glick
Jesse Glick

Reputation: 25451

A utility I wrote called jkillthread can be used to stop any thread in any Java process, so long as you can log in to the machine running the service under the same account.

Upvotes: 0

Coulemelle
Coulemelle

Reputation: 361

I guess it is too late to answer but my help some people.

  1. Install the monitoring plugin. (http://wiki.jenkins-ci.org/display/JENKINS/Monitoring)
  2. Go to jenkinsUrl/monitoring/nodes
  3. Go to the Threads section at the bottom
  4. Click on the details button on the left of the master
  5. Sort by User time (ms)
  6. Then look at the name of the thread, you will have the name and number of the build
  7. Kill it

I don't have enough reputation to post images sorry.

Hope it can help

Upvotes: 11

cheffe
cheffe

Reputation: 9500

I use the Monitoring Plugin for this task. After the installation of the plugin

  1. Go to Manage Jenkins > Monitoring of Hudson/Jenkins master
  2. Expand the Details of Threads, the small blue link on the right side

  3. Search for the Job Name that is hung

    The Thread's name will start like this

    Executor #2 for master : executing <your-job-name> #<build-number>

  4. Click the red, round button on the very right in the table of the line your desired job has

Upvotes: 28

Ernie
Ernie

Reputation: 567

Have had the same problem happen to me twice now, the only fix sofa has been to restart the tomcat server and restart the build.

Upvotes: 0

blokkie
blokkie

Reputation: 5565

I've looked at the Jenkins source and it appears that what I'm trying to do is impossible, because stopping a job appears to be done via a Thread interrupt. I have no idea why the job is hanging though..

Edit:

Possible reasons for unstoppable jobs:

  • if Jenkins is stuck in an infinite loop, it can never be aborted.
  • if Jenkins is doing a network or file I/O within the Java VM (such as lengthy file copy or SVN update), it cannot be aborted.

Upvotes: 8

Related Questions