CODEWITHSUNDEEP
gitjenkinsgitlab
Christian Dechery
Christian Dechery

Reputation: 880

Jenkins fails to abort job on git connection failure

I have a job setup in my Jenkins which copies a git repository (on the web), to our local (intranet) git. It works 99% of the time with no problems.

But every now and then, since our internet connection is not that great, it fails to connect to the remote repository. The problem isn't exactly in the connection issue, which is a given. The problem lies in how Jenkins handles it.

It is now set to run every 15min looking for changes in the remote repo. (We didn' manage to set a webhook yet) Nine times out of ten, there won't be and it ends with "No changes - OK".

When the internet connection is not that good, the query on git will timeout, but instead of aborting the job, and running it again 15min later, it will go crazy and try to download the entire repo, which is almost 200mb in size, and of course it will crash becasue - well, the connection at that point is not good. Worst part? The job will not run again, until I manually do so.

Any hints on how to fix this behaviour? Below is a log showing Jenkins weird behaviour.

16:30:09  > git fetch --tags --progress https://[email protected]/yyy/mmmmm.git +refs/heads/*:refs/remotes/origin/*
16:40:09 ERROR: Timeout after 10 minutes
16:40:09 ERROR: Error cloning remote repo 'origin'
16:40:10 hudson.plugins.git.GitException: Command "git fetch --tags --progress https://[email protected]/yyy/mmmmm.git +refs/heads/*:refs/remotes/origin/*" returned status code 143:
16:40:10 stdout: 
16:40:10 stderr: remote: Counting objects: 45507, done.        
16:40:10 remote: Compressing objects:   0% (1/397)           
remote: Compressing objects:   1% (4/397)           
remote: Compressing objects:   2% (8/397)           
remote: Compressing objects:   3% (12/397)           
remote: Compressing objects:   4% (16/397)        
[...]
Receiving objects:  54% (24849/45507), 83.98 MiB | 82.00 KiB/s   
Receiving objects:  54% (24849/45507), 84.04 MiB | 84.00 KiB/s   
16:40:10    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1799)
16:40:10    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1525)
16:40:10    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:65)

Upvotes: 1

Views: 722

Answers (1)

user8399477
user8399477

Reputation:

This might not be the most glorious solution, but setting the Retry Count under Advanced Project Options might alleviate some of your woes. If Retry Count is set, Jenkins will retry checking out the code from the repository a certain number of times (whatever it is set to) when a build fails.

If the Jenkins server is actually crashing, which it sounds like it might be doing from the details you provided, then I would suggest implementing a webhook right away, and do manual builds until the process is remedied. There's nothing wrong with doing manual builds for a short period of time. It's still an acceptable CI practice, albeit full automation is ideal.

Upvotes: 1

Related Questions