CODEWITHSUNDEEP
bashjenkinsjenkins-pipeline
sorin
sorin

Reputation: 170846

How do I limit the amount of job console log on Jenkins?

From time to time I encounter jobs that end-up putting Jenkins down because they log too much to the console, sometimes due to an more or less endless loop or too much verbosity.

I am looking for a solution that would prevent/stop/kill spamming jobs.

We are using Jenkins pipelines and all these occasional spamming jobs are inside sh(), which makes me think that maybe we should add some kind of smart wrapper.

Please, do not suggest fixing the spamming source instead. I asked this question because I want to prevent this kind of issue from outside the potential spammer script, that's because we have hundreds of jobs and because is impossible to avoid a sooner or later spamming bug caused by a developer mistake.

Shortly: we cannot control what is run inside the shell script, but we can control how the shell is run.

Upvotes: 3

Views: 5468

Answers (3)

alexandrosdelta
alexandrosdelta

Reputation: 31

There is a plugin that stops jobs if their console log has exceeded a defined size limit.

It is called "Build log file size checker"

If your normally successful job logs are consistent in size, and do not vary hugely, then this could help you distinguish the looping or spamming ones.

Upvotes: 3

sorin
sorin

Reputation: 170846

After many hours wasted with various attempts to address this I was able to come up with a working solution that looks like:

def sh2(script) {
    sh '''( ''' + script + ''' ) 2>&1 | tee output.log | awk -v offset=${MAX_LINES:-200} '{ if (NR <= offset) print; else { a[NR] = $0; delete a[NR-offset] } } END { for (i=NR-offset+1; i<=NR; i++) { if (!match(a[i],"^[[:space:]]*$")) print a[i]} }
}

You can find the latest version at https://github.com/ssbarnea/rhos-ci/blob/master/vars/sh2.groovy in case I will find other bugs or improvements.

As you can see it will limit both head and tail to 200 lines if MAX_LINES is not defined. If defined it will be used.

This solution is the only one that does not repeat output when the total number of lines is <200. Also it does work in streaming mode, so you do see the output in real time.

Note: the linked version has extra features, like ability to log full output to auto-sequenced log files.

Upvotes: 1

Spencer Malone
Spencer Malone

Reputation: 1509

You can add the returnStdout variable to your sh calls like so:

node(){
    // Eliminates returned output (you can also redirect this to a variable!)
    sh returnStdout: true, script: 'echo hi; echo hi; echo hi; echo hi;'
    // Has the normal output
    sh script: 'echo hi; echo hi; echo hi; echo hi;'
}

Which changes to the output like so:

[Pipeline] node
Running on ***
[Pipeline] {
[Pipeline] sh
[***] Running shell script
+ echo hi
+ echo hi
+ echo hi
+ echo hi
[Pipeline] sh
[***] Running shell script
+ echo hi
hi
+ echo hi
hi
+ echo hi
hi
+ echo hi
hi
[Pipeline] }
[Pipeline] // node

If you need to automate this so that everybody does this by default, you should look into building a Groovy Shared Library that is auto loaded in your Jenkins instance (I recommend this for more large users anyways!). Once you have that set up, make a new script for executing shell steps and ask users to use that instead of the normal sh step

Here's a good starting doc for that stuff: https://jenkins.io/doc/book/pipeline/shared-libraries/

Upvotes: 0

Related Questions