How to checkout a branch only listed in `git ls-remote`?

Question

I got a situation that I'm unable to switch to a branch only listed in git ls-remote, here is the detail:

I forked an github repoA as repoB, created and pushed my own branches to the repoB in ComputerA, and in ComputerB I clone my forked repo into local disk, added the remote upstream, and tried to switch to the branches I created but failed, I can successfully switch to the same branch in github web page though.

The following result is from repoB in ComputerB.

ls-remote branches:

$ git ls-remote --heads
2da2080ea7201fc7928e947dc3214dd89d86c4ba        refs/heads/enable-vim-better-whitespace
433cedd84bba8bcdf3584734906b2c0fd3b6dc3a        refs/heads/fix-lsp-cache-dir
ff65e1cd687d0c144e98b09e4d7a164f8b6bfd3e        refs/heads/gh-pages
17e53cf01badebc2abef7df375903da71bf884d8        refs/heads/master
7b8f8a2dccb0715ff1c1c411abf40b2ff6cec30b        refs/heads/vim-plug
26b8a0ba594af1068997c70c4ef0f503571557b3        refs/heads/vundle

list branches:

$ git branch
  abc
* master

$ git branch -r
  origin/HEAD -> origin/master
  origin/master
  upstream/gh-pages
  upstream/master
  upstream/vim-plug
  upstream/vundle

$ git branch -a
  abc
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/upstream/gh-pages
  remotes/upstream/master
  remotes/upstream/vim-plug
  remotes/upstream/vundle

The branch abc is a local branch I haven't pushed yet.

and I tried several methods to switch to branch such as fix-lsp-cache-dir like

$ git checkout fix-lsp-cache-dir
error: pathspec 'fix-lsp-cache-dir' did not match any file(s) known to gi

$ git checkout -t origin/fix-lsp-cache-dir
fatal: 'origin/fix-lsp-cache-dir' is not a commit and a branch 'fix-lsp-cache-dir' cannot be created from it

I tried google, but all of the suggested methods failed.

So what can I do to switch to the branch only list in git ls-remote

torek · Accepted Answer

You mentioned in a comment that you have multiple remotes, origin and upstream. This interferes with—well, may interfere with—a Git feature that people often don't know they're depending on: git checkout's so-called DWIM mode. That's not the problem yet, but we might as well address it (in the long section below).

You mentioned in a second comment that git config -l includes this output:

remote.origin.fetch=+refs/heads/master:refs/remotes/origin/master

This is not the normal setting for a typical standard clone with an origin. The normal setting is:

remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*

The setting you have is the norm if you originally ran git clone --single-branch, or git clone --depth=... (which implies --single-branch).

To make things work conveniently, you will need to change, or add to, your remote.origin.fetch setting. For instance, if you first change it to +refs/heads/*:refs/remotes/origin/* (see VonC's updated answer), you can then run:

git fetch origin

followed by:

git checkout -t origin/fix-lsp-cache-dir

or even just:

git checkout fix-lsp-cache-dir

This shortest method will always work if you have just the one remote origin. It will sometimes fail if you have more than one remote, in which case, you need to use the either the slightly longer git checkout -t origin/fix-lsp-cache-dir, or a separate git branch command, to create your own branch name fix-lsp-cache-dir.

No matter what, you will need one git fetch that fetches from origin first. You can name origin explicitly in your git fetch, or use one of the options that fetches from all remotes (git fetch --all or git remote update, though using git remote strays into new territory that brings in many new options).

Long: what's going on, behind the scenes

To make sense out of all of this, you need to know all about:

branch names, with which you're already familiar but are internally stored with refs/heads/ stuck on the front (as you're seeing with git ls-remote);
remote-tracking names—Git calls this remote-tracking branch names but they're not actually branch names, so I prefer to drop that word from the middle: they are internally stored with refs/remotes/ stuck on the front, followed by the remote name itself;
remotes, which are short strings like origin and upstream that, if nothing else—usually there's something else too—store a URL;
refs or references, which are the long forms of branch names, tag names (refs/tags/*), remote-tracking names, and other less-common names like refs/notes/* and refs/stash;
refspecs, which are mostly just pairs of refs separated by a colon : and optionally prefixed by a plus sign +; and last,
git checkout's "DWIM mode" feature. DWIM stands for Do What I Mean (as opposed to what I typed in). This particular acronym goes back to Xerox PARC and Warren Teitelman: see Eric Raymond's Jargon File entry and the Wikipedia article on Teitelman.

Refs, refspecs, and remotes

Really, you already know about refs. They're just the full names of each of the various kinds of reference. They let commands like git fetch know whether they are dealing with a branch name (refs/heads/master) or a remote-tracking name (refs/remotes/origin/master) or whatever, if they even care.¹

The simplest form of refspec is just a pair of refs with a colon. The name on the left is a source and the name on the right is a destination. For git fetch, the source part means: Using the same stuff you saw in git ls-remote output, find the name and value in the repository I'm fetching from. The destination part means create or update the destination name in my own repository.

The leading plus sign, if it appears, sets the --force flag for any updates that occur because of that refspec. Hence:

+refs/heads/master:refs/remotes/origin/master

is a refspec saying: Grab their master branch, and use that to create or update my origin/master remote-tracking name. Force this update if needed. You'll get any new commits they have on their master, then create-or-update your origin/master. You'll do this update to your own origin/master even if that means some commits "fall off" your origin/master in the process (--force).

I mentioned that remotes pack a bit more than just a URL. Each remote lists some number of default fetch refspecs. Usually it's just one, but usually that one is:

+refs/heads/*:refs/remotes//*

with the remote part filled in. This particular refspec says: Take all of their branch names—all the strings that match refs/heads/*—and create or update, forcibly, all of my corresponding remote-tracking names. The corresponding names for remote origin are refs/remotes/origin/*, so that's what appears here.

Single-branch clones work by the simple expedient of using the single branch name in the refspec. Now your git fetch doesn't create or update the rest of your potential remote-tracking names. Fix that, and your git fetch will create or update the rest of your remote-tracking names.

Note that using refs/heads/* enables one more feature: --prune. Add --prune to your git fetch command—or set fetch.prune to true in your configuration—and git fetch will not only create or update the right set of remote-tracking names, but also delete any left-over remote-tracking names that no longer have a source.

For instance, if the Git on origin has a branch named X for a short while, and you run git fetch, your Git creates your own origin/X. But then whoever controls the Git on origin removes branch X. If you don't have pruning enabled, you continue to carry origin/X: your Git created and updated it while it exists, but now that it doesn't, your Git does nothing about that. Enable pruning, and your Git says to itself: Aha, I have a leftover junk origin/X! I'll snip it off automatically. Prune should probably be the default, with a "don't-prune" option, but it isn't.

¹Fetch actually does care, because of a bunch of magic weird stuff it tries to do with tags.

Checkout's "DWIM mode", and when and why it fails with two or more remotes

When you first clone a Git repository (without --single-branch), your own Git gets remote-tracking names for every branch in the origin repository:

git clone https://github.com/git/git/

gives you five remote-tracking names for the five branches in the Git repository over on GitHub, for instance.

As the last step of this git clone, your Git effectively² runs git checkout master. At this stage you don't have a branch named master. In fact, you have no branch names at all! So how can git checkout check it out? How can:

git checkout

ever work, when there are no branch names at all?

The answer is that git checkout actually creates your branch name master. See the sidebar below as well (formatted as an extra section since I can't do real sidebars). When git checkout is given what seems like it could be a branch name, but isn't, it looks at all of your remote-tracking names: origin/master, origin/maint, origin/next, and so on, if you are using the Git repository for Git, for instance. If exactly one name matches, then your Git acts as though you actually ran:

git checkout -t origin/

which tells git checkout: Create the branch, setting the remote-tracking name as its upstream. Now that the name exists, now git checkout can check it out.

This process fails if there are two or more matching names. Suppose, for instance, that you don't have fix-lsp-cache-dir as a branch name, but you do have, in your own Git repository, origin/fix-lsp-cache-dir and upstream/fix-lsp-cache-dir. You run:

git checkout fix-lsp-cache-dir

which doesn't find fix-lsp-cache-dir but does find origin/fix-lsp-cache-dir and upstream/fix-lsp-cache-dir. It found not one but two remote-tracking names. Should it use the origin one, or the upstream one? It doesn't know.

At this point, git checkout simply gives up and says that it has no idea what you mean by fix-lsp-cache-dir. So now you need, e.g., git checkout -t origin/fix-lsp-cache-dir, which is an explicit instruction: Look up the remote-tracking name origin/fix-lsp-cache-dir, use that to create fix-lsp-cache-dir, then check out fix-lsp-cache-dir. That supplies the answer about which upstream remote-tracking name to use, and at the same time, what branch name to create.

²I say "effectively" here because the code inside git clone that does this, does not literally run git checkout, nor bother with a lot of the DWIM mode stuff: it knows exactly what it's put into the repository already and can cheat. If you split your git clone into a series of separate commands:

git init
git remote add origin 
git fetch
git checkout master

you will literally run git checkout master and invoke the DWIM mode I'm describing.

(Mental exercise: compare and contrast Git's branch DWIM and your smartphone's autocorrect.)

Extra-long sidebar: how Git branches really work

Every Git branch name—in fact, every Git reference—really just stores one hash ID. For branch names—and by implication, remote-tracking names—the hash ID is constrained to be a commit hash ID; some other refs have more flexibility, e.g., tag names can point to any of Git's four internal object types.

The thing is, when we say "branch master", or "this commit is on branch master", or anything along those lines, we often don't mean one particular commit, even though the actual branch name master is only able to identify one particular commit. How this works explains a lot about Git.

In capsule form:

To create a branch, we write the hash ID of some existing, valid commit into a name that did not exist before.
To update a branch, we write the hash ID of some existing, valid commit into a name that already existed. It no longer identifies the commit it remembered a moment ago. Now it identifies the one we picked instead.

No matter what, though, we start with a commit hash ID. So in a sense, it's the commits that matter, not the branch names (though of course we want those too!).

In Git, every commit is identified by its own unique, big ugly hash ID. For instance, one commit in the Git repository for Git is 9c9b961d7eb15fb583a2a812088713a68a85f1c0. (This is a commit that is in preparation for Git version 2.23, but is not any particular release.) These hash IDs are fine for Git to use—it's a computer program, and it won't make mistakes in using these things as keys in a key-value database—but they're pretty useless to mere humans. We do better with names, like master. If we create our branch name master and make that name mean "commit 9c9b961d7eb15fb583a2a812088713a68a85f1c0", we can run:

git log master

or:

git diff my-branch master

or whatever. The name master will pick out commit 9c9b961d7eb15fb583a2a812088713a68a85f1c0 each time. But then how does Git know that commit 8619522ad1670ea82c0895f2bfe6c75e06df32e7—another random-looking hash ID—is the commit that comes right before master (9c9b961d7eb15fb583a2a812088713a68a85f1c0)?

The answer is that 8619522ad1670ea82c0895f2bfe6c75e06df32e7 is stored inside 9c9b961d7eb15fb583a2a812088713a68a85f1c0:

$ git cat-file -p 9c9b961d7eb15fb583a2a812088713a68a85f1c0 | sed 's/@/ /'
tree 33bba5e893986797fd68c4515bfafd709c6f69e5
parent 8619522ad1670ea82c0895f2bfe6c75e06df32e7
author Junio C Hamano  1563561263 -0700
committer Junio C Hamano  1563561263 -0700

The sixth batch

Signed-off-by: Junio C Hamano

The parent line here gives the raw hash ID of the previous commit.

Every Git commit—well, almost every one—has at least one parent.³ Git can move one step backwards in history, from a commit to its parent. The parent itself has another parent, so Git can move one more step. The path obtained by moving, step by step, from commit to parent, is the history in Git repository.

We can draw this, for simple linear chains, by pretending for a moment that instead of big ugly hash IDs, Git uses one letter names for each commit:

... <-F <-G <-H   <--master

The last commit in the chain is commit H. That's the hash ID stored under the name master. We say that master points to H. H in turn stores the hash ID for G, so we say that H points to G. G stores the hash ID for F, so G points to F. F points to F's parent. This continues all the way down the line until we hit a commit that doesn't have a parent, such as the first commit ever for this repository ... and those are the commits that are "on" branch master.

To add a new commit, we have Git save a snapshot of all of our source files, add our name and email address and other stuff that git log shows, use the actual hash ID of commit H as the parent, and write out a new commit. This new commit gets a new, unique hash ID, but we'll just call it I. Then Git simply overwrites the name master with this new hash ID:

... <-F <-G <-H <-I   <--master

and the master branch is now one commit longer. The last commit in the chain is called the tip commit. We know—or find—the tip commits in a Git repository by reading the hash IDs from the branch names.

The branch name master simply identifies the last commit in the chain. The various Git commands that move branch names, or remote-tracking names, around, such as git reset or git branch -f or—for remote-tracking names—git fetch—are really just making the names point to one specific commit.

If we can start at the new tip, and use the internal, backwards-pointing arrows to find the old tip, then all we have done is add some commit(s) to the branch. When we use git commit to create a commit, that's just what it does: it creates one new commit, which becomes the tip, and which has the old tip as its parent.

When we use git fetch and we get, say, three or five new commits for our remote-tracking name origin/master, the last of these—the tip—leads back, eventually, to where our origin/master pointed before we ran git fetch. So the new commits are just newly added to the origin/master remote-tracking name.

Git calls this kind of name update, that only adds things, a fast-forward. You can do fast-forwards with git fetch, updating your remote-tracking names, and with git push, giving new commits to some other Git and having them update their branch names. In both cases, your Git and/or their Git have not lost any commits, because starting at the new tip and working backwards, you or they arrive at the old tip.

You can also—with a few extra wrinkles—do a fast-forward with git merge. If git merge does a fast-forward instead of a merge, it's used commits that you already have, without actually making any new commits. For instance, after git fetch origin, you might have:

...--F--G--H   <-- master (HEAD)
            \
             I--J   <-- origin/master

Here you actually are on your own master, indicated by attaching the special name HEAD to the name master. Your Git can now do a fast-forward not-really-a-merge by moving the name master so that it points to commit J, and doing a git checkout of commit J, all at the same time:

...--F--G--H--I--J   <-- master (HEAD), origin/master

That's what a fast-forward merge is: it's really not a merge at all, but just a git checkout that also drags the current branch name forward, the same way that git fetch fast-forwarded your origin/master a moment ago.

The --force flag is needed when the operation is not a fast-forward. For instance, suppose you just did the above, so now master and origin/master both identify commit J. Meanwhile, whoever controls the repository over at origin says: Oh crap! Commit J is bad! I'm throwing it out with git reset --hard and adding a new commit K instead! Now you run git fetch again and get:

          K   <-- origin/master
         /
...--H--I--J   <-- master (HEAD)

You still have commit J: it's on your master. They tried to toss out commit J (whatever its actual hash ID is—your Git and their Git agree on its hash ID). Your origin/master now points to K, and K's parent is I, not J. Your origin/master was just force-updated.

You'll see this in git fetch output:

$ git fetch
...
 + a83509d9fc...0ddebcb508 pu          -> origin/pu  (forced update)

The pu branch, in the Git repository for Git, is one that everyone agrees gets force-updated regularly. So my origin/pu used to identify a83509d9fc, but now it identifies 0ddebcb508. Note that +, the words (forced update), and the fact that there are three, not two, dots between the two hash IDs: those are the three ways that git fetch announces that my origin/pu has just been force-updated. I can now do this:

$ git rev-list --left-right --count a83509d9fc...0ddebcb508
79  214

which tells me 79 commits were dropped (from my old origin/pu) and 214 commits were added (to my new updated origin/pu). I don't actually care in this case, but if I did for some reason, I can see what they did over at origin.

(Slightly more useful:

$ git rev-list --count master..origin/master
210

tells me that there are 210 new commits that I can bring into my master now. To actually look at those commits, I probably want git log.)

³A commit with no parents is defined as a root commit. That's the kind of commit you make when you make the very first commit in a new, totally-empty Git repository. This first commit can't have a parent, so it doesn't.

A commit with two or more parents is defined as a merge commit. That's the kind of commit that git merge typically makes. The first parent is business as usual; any additional parents tell Git which commit(s) were merged.

How to checkout a branch only listed in `git ls-remote`?

Answers (2)

Long: what's going on, behind the scenes

Refs, refspecs, and remotes

Checkout's "DWIM mode", and when and why it fails with two or more remotes

Extra-long sidebar: how Git branches really work

Related Questions