How can I fix Jenkins error due to bad remote git repo

Question

After trying to run Jenkins tests on a GitHub Pull Request, I'm running into the following error:

error: some local refs could not be updated; try running
00:07:44  'git remote prune git@github.com:myrepo/myrepo.git' to remove any old, conflicting branches

Running any or all of the following does not fix it:

• git remote prune git@github.com:myrepo/myrepo.git
• git remote prune origin
• git fetch origin -t --prune
• git gc --prune=now

Jenkins is running this command in its workspace:

git fetch --tags --progress origin  +refs/pull/*:refs/remotes/origin/pr/*

Which shows me that there is a bad ref on the remote:

git fetch --tags --progress origin  +refs/pul
l/*:refs/remotes/origin/pr/*
Warning: Permanently added the RSA host key for IP address '192.30.252.131' to the list of known hosts.
error: 'refs/remotes/origin/pr/2730' exists; cannot create 'refs/remotes/origin/pr/2730/head'
From github.com:myrepo/myrepo
 ! [new branch]      refs/pull/2730/head -> origin/pr/2730/head  (unable to update local ref)
error: some local refs could not be updated; try running
 'git remote prune origin' to remove any old, conflicting branches

There is a ticket closed as Won't Fix on Jenkins about a similar error: JENKINS-19591

I've tried deleting the remote ref:

git push origin :refs/origin/pr/2730/head
Warning: Permanently added the RSA host key for IP address '192.30.252.128' to the list of known hosts.
remote: warning: Deleting a non-existent ref.
To git@github.com:myrepo/myrepo.git
- [deleted]        refs/origin/pr/2730/head

I still run into the same error afterwards :-(

How can I fix the remote Pull Request ref?

After checking in .git/logs/refs/remotes/origin/pr/ I found that the branch it is complaining about is the only one which is a normal file:

$ ls -ld .git/logs/refs/remotes/origin/pr/273*
drwxrwxr-x 2 jenkins jenkins 4096 Mar 21 08:05 .git/logs/refs/remotes/origin/pr/273
-rw-rw-r-- 1 jenkins jenkins  248 Mar 21 08:05 .git/logs/refs/remotes/origin/pr/2730
drwxrwxr-x 2 jenkins jenkins 4096 Mar 21 08:05 .git/logs/refs/remotes/origin/pr/2731
drwxrwxr-x 2 jenkins jenkins 4096 Mar 21 08:05 .git/logs/refs/remotes/origin/pr/2732
drwxrwxr-x 2 jenkins jenkins 4096 Mar 21 08:05 .git/logs/refs/remotes/origin/pr/2733
drwxrwxr-x 2 jenkins jenkins 4096 Mar 21 08:05 .git/logs/refs/remotes/origin/pr/2734
drwxrwxr-x 2 jenkins jenkins 4096 Mar 21 08:05 .git/logs/refs/remotes/origin/pr/2735


$ file .git/logs/refs/remotes/origin/pr/273*
.git/logs/refs/remotes/origin/pr/273:  directory
.git/logs/refs/remotes/origin/pr/2730: ASCII text
.git/logs/refs/remotes/origin/pr/2731: directory
.git/logs/refs/remotes/origin/pr/2732: directory
.git/logs/refs/remotes/origin/pr/2733: directory
.git/logs/refs/remotes/origin/pr/2734: directory
.git/logs/refs/remotes/origin/pr/2735: directory
.git/logs/refs/remotes/origin/pr/2736: directory
.git/logs/refs/remotes/origin/pr/2737: directory
.git/logs/refs/remotes/origin/pr/2738: directory
.git/logs/refs/remotes/origin/pr/2739: directory

$ find .git/logs/refs/remotes/origin/pr/ -maxdepth 1 -type f
.git/logs/refs/remotes/origin/pr/2730

$ cat .git/logs/refs/remotes/origin/pr/2730
0000000000000000000000000000000000000000 0c960f4b8bf30109536ec1b7a1d54a636e21fd87 Jenkins <ops@tangogroup.com> 1395389111 +0000 fetch --tags --progress git@github.com:TangoGroup/program_creator.git +refs/heads/*:refs/remotes/origin/*: storing head

The contents look a bit strange... not sure whether that's how it is supposed to be...

Answer 1

I want to leave a comment here because I encountered issue with same symptoms but with a different root cause. Our Jenkins was running git fetch --tags --progress -- git@url_of_the_repo.git +refs/pull/*:refs/remotes/origin/pr/* +refs/heads/*:refs/remotes/origin/* and every PR was outputted as and error

 ! [new branch]      refs/pull/1/head -> origin/pr/1/head  (unable to update local ref)
error: some local refs could not be updated; try running

The root cause for it was an existing remote branch named pr. Once the branch was deleted the issue went away.

Answer 2

Unfortunately I can't add a comment to the answer (which I think is the appropriate answer). But we encountered this problem on our Jenkins instance yesterday and it had a nuance so I wanted to leave that for future folks in this situation.

For us, deleting the both the 'merge' and 'head' refs did not solve the problem. (It's possible this is because it was based on a pull request coming in from a fork that is not under our control. Note that the branch on that fork was not corrupt.)

Likewise, we attempted to delete the entire PR on the remote reference (basically copy/pasting the answer on this page, and changing the PR #), and that did not solve the problem either. (It deleted cleanly, but quickly re-appeared after I pruned and refetched; it was as if our delete had no effect on the remote.) This is because the pulls reference is read-only.

Oddly, the merge hash will change whenever there is a merge to master, but that remained corrupted for us even when we had merges on our master branch.

Contacting Github support with a clear explanation of the offending branch and the error gave them what they needed to resolve the problem. (From my end, I saw a force-push across all the PRs' merge and head refs.) Obviously you'll want to clearly understand the problem before contacting them, but in our case it was truly on a remote branch that we had no control over.

Answer 3

Try running

git push origin :refs/remotes/origin/pr/2730/merge 
git push origin :refs/remotes/origin/pr/2730/head 

to remove the remote PR branches and rerun your Jenkins job.

Answer 4

Judging by this part:

error: 'refs/remotes/origin/pr/2730' exists; cannot create 'refs/remotes/origin/pr/2730/head'

It seems your Jenkins job has a dirty workspace: it's as if the .git/logs/refs/remotes/origin/pr/2730/head file exists but it cannot replace it. Not to mention, it's strange that it's trying to create it even though it doesn't exist in your remote.

Do you mind wiping out your workspace? Jenkins has a menu option for that. After you do that, make sure the workspace directory is really deleted. Confirm that by looking directly in the filesystem, if you can.

UPDATE

As you can reproduce the problem by running git fetch --tags --progress origin +refs/pull/*:refs/remotes/origin/pr/* on another PC, it's evidently not a workspace problem. I think this other question explains well what's going on.