Git push old version of a "--skip-worktree" file

Question

I'm using Git to transfer code from a DEV machine to a PROD machine. Most of the code is common, but I have specific configuration files that need to be different between the two.

I want my git to represent the current state of the PROD machine. The idea is to push most files from DEV, and only specific local_config_file from PROD. Therefore I executed the following command on DEV so that the DEV local_config_file would not impact, or be impacted by GIT:

git update-index --skip-worktree local_config_file

However, when I try to push code from DEV to git I get this error :

error: failed to push some refs to [my_git]
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Merge the remote changes (e.g. 'git pull')
hint: before pushing again. 

And when I run git diff origin/branchName branchName (still on DEV), it only shows me differences from files I want to push (that's ok ...) and differences from an old version of the local_config_file, despite being set as skip-worktree.

For experimental purpose, I tried to push force into git and end up with the old version of my local_config_file in git ...

Anyone knows how to solve this ? Thanks for your help

Answer 1

I want my git to represent the current state of the PROD machine.

OK.

Remember that Git stores commits, not files. Commits store files—in fact, each commit contains a full snapshot of every file—but Git works one commit at a time. Each Git repository is, at its heart, a collection of commits, and each repository either has some commit, or lacks it. The commit is thus all-or-nothing: all of the files, or else you don't have the commit at all.

The idea is to push most files from DEV ...

You cannot do this in Git. You must push commits.

... and only specific local_config_file from PROD. Therefore I executed the following command on DEV so that the DEV local_config_file would not impact, or be impacted by GIT:

git update-index --skip-worktree local_config_file

This does not do what I think you think it does. That's not the immediate source of the next problem, but it is something you will need to clear up before you can get what you want.

error: failed to push some refs to [my_git]
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Merge the remote changes (e.g. 'git pull')
hint: before pushing again.

This, on the other hand, means more or less what it says. Its advice may be slightly wrong for your particular situation, though.

What to know about commits and Git's index

When using Git, remember these things:

• Git is all about the commits. Branches—more specifically branch names—are useful to you (and to Git to get it to help you) but Git is really about the commits.

• Every commit has a unique number. This number is a random-looking (but not at all random) hash ID, rather than a nice simple counting number, but they are still numbered. This lets any two Gits talk to each other and agree as to whether they have, or lack, some particular commit, just by comparing the commit numbers.

• Each commit has two parts: a snapshot (of all files) stored in a read-only, Git-only, frozen-for-all-time but de-duplicated format—this saves lots of space since most files mostly just get re-used in the next commit—plus some metadata, i.e., information about the commit itself. The metadata include things like who made the commit, when, and why (log message). This also includes, though, the commit number of the previous commit.

Because the files inside a commit are frozen, de-duplicated, and Git-only, they have to be extracted out of a commit for you to use them. The place where these files are defrosted and decompressed—reconstituted or rehydrated for your use—is called your working tree or work-tree. You pick some commit—ultimately, by its internal commit number—and ask Git to extract that commit. Git does so, and now you have all the files from that commit.

In this sense, you have two copies of every file: the current commit has the frozen and de-duplicated Git-only copy, and your work-tree has the ordinary format copy that you can work with.

Here's where Git gets a bit tricky, though: to make a new commit, Git doesn't use your work-tree copy! That copy is yours, not Git's. Instead, Git sneakily keeps a third copy. It's not exactly a copy, because it's pre-de-duplicated, but it acts like one. This third copy is in what Git calls, variously, the index, or the staging area, or sometimes—rarely these days—the cache.

When you first check out a commit, Git fills in its index from that commit, and then fills your work-tree from the files from that commit too. If you change your work-tree copy of some file, you generally want to use git add, which tells Git: Copy the work-tree copy back into the index, replacing the copy that was there before. That way, the index always holds your proposed next commit. (The files in the index are kept in the frozen and de-duplicated format, ready to go, making the next commit go fast, too. The git add step does the compressing and de-duplicating.)

What skip-worktree does

What the --skip-worktree bit does is tell Git: If my work-tree copy doesn't match your index copy, that's OK. Don't say anything about it. Don't use the work-tree copy to update the index copy either. Just skip right over the work-tree copy. That way, your proposed next commit keeps the index copy that you extracted from some existing commit.

When you use git checkout to switch to some other existing commit, though, Git will still copy the frozen-format file into the index (and sometimes into your work-tree too: these details vary with the use of Git's sparse checkout mode). The --skip-worktree bit just means it won't complain when you change the work-tree copy, and git add won't update the index copy.

It's time to bring this part back to your particular issue:

I want [to make a new commit] to represent the current state of the PROD machine.

Git will make a new commit from the files that are in Git's index, not from the files in your work-tree. So, to do this, you must load your own Git's index with the correct files.

The idea is to [use] most[ly] files from DEV and only specific local_config_file from PROD ...

So, to do this, you should make sure that Git's index copy of this local config file matches the one on PROD. If it doesn't:

1. move the local config file out of the way;
2. clear the --skip-worktree flag (git update-index --no-skip-worktree local_config_file);
3. install the PROD config file as the local config file;
4. use git add to copy the correct file into Git's index;
5. feel free to switch the config back and/or set flags again, now that the index copy matches the file you'd like to have in the next commit.

There are several alternatives to this process. For instance, a pretty standard way to deal with this is to be sure never to store any actual configuration file in Git at all. Store only sample or default configurations; for a real, live configuration, use the sample or default to whatever extent it's useful, but add whatever customization is required, then don't have Git manage that file at all. (Unfortunately, having put the configuration into Git, any update that removes the file from Git will cause a Git update step to remove the file from the work-tree. So at this point, unless you're willing to go through a painful switchover, you're a bit stuck here.)

How to see what's actually in an index file

When you have set the --skip-worktree flag set on the index copy of a file, it's hard to know what's actually in the index copy, because Git doesn't show you the file directly. Git normally just shows you whether the index copy matches the work-tree copy. Setting --skip-worktree tells Git: don't bother comparing (index and work-tree copies) as well as suppressing updates.

You can, however, view the index contents directly:

$ echo 'skip this in worktree' > file
$ git add file
$ git commit -m 'add file to skip'
[master 7cee93e] add file to skip
 1 file changed, 1 insertion(+)
 create mode 100644 file
$ git update-index --skip-worktree file
$ ls
file    README
$ git cat-file -p :file
skip this in worktree
$ echo more >> file
$ git status
On branch master
nothing to commit, working tree clean
$ cat file
skip this in worktree
more
$ git add -f file
$ git cat-file -p :file
skip this in worktree
$ 

The above shows how, despite my changes to file file, git status and git add leave the index copy alone. But I can view the index copy with git cat-file -p :file. The leading colon : here means let me see the copy that is in the index.

If I unset the --skip-worktree flag, git status starts saying more:

$ git update-index --no-skip-worktree file
$ git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   file

no changes added to commit (use "git add" and/or "git commit -a")

Again, all that --skip-worktree does is make sure that git status doesn't complain and git add doesn't actually add. The index copy of file file is still there, and still goes into new commits I make.

the tip of your current branch is behind its remote counterpart

Your last problem is here. Remember how, as mentioned above, each commit has two parts:

• The data in a commit is a snapshot of all files.
• The metadata in a commit is information about the commit, including author and committer (name and email addresses), date-and-time-stamps, and so on.

In the metadata, each commit stores the unique number of its immediate predecessor commit (or for a merge commit, predecessor commits, plural). That is, given some commit, Git can find the commit that comes before that commit. Git calls this stored hash ID the parent of the commit.

Git cannot go forward in time with this information. Git can only go backwards. But that is sufficient: given the commit number—the hash ID—of the latest commit, Git can work backwards from there, to earlier commits.

These are the actual branches in Git. The names—the branch names like master and develop—are just a trick: each name stores the random-looking hash ID of the last commit that is to be considered part of that branch. This means that it's the commits themselves that matter. The names just let you, or Git, get started.

We can draw this situation like this:

... <-F <-G <-H

where H is the hash ID of the last commit in some chain of commits. To find H's hash ID quickly, Git has a name, like master, store that hash ID:

...--F--G--H   <-- master

Using that hash ID, Git can extract commit H, or, if you ask it to, use H's stored parent hash ID—the one for G—go back in time one commit to commit G and extract that commit. Having found commit G, Git can use its stored parent hash ID, which locates F, to extract commit F. Having found that commit, Git can go back even further, if and when necessary.

To make a new commit that comes after commit H, Git:

1. gathers any necessary metadata: your name, your email address, and so on;
2. gets H's hash ID (it's right there handy in the name master);
3. freezes this, plus all of the index copies of all files, into a new commit, which we'll call I; and last, in a sneaky but clever move
4. writes I's new hash ID into the name master.

This means we go from:

...--F--G--H   <-- master

to:

...--F--G--H   <-- master
            \
             I

to:

...--F--G--H--I   <-- master

and now we have a new commit on master, that simply adds on to the chain.

To send this new commit to some other Git, we have our Git call up their Git, believing that their Git also has:

...--F--G--H   <-- master

in its set of commits and its branch names. We have our Git send them our new commit I, so that they now have:

...--F--G--H   <-- master
            \
             I

in their repository. Then we have our Git ask them: If it's OK, please set your master so that it refers to commit I now.

When they say no, and add:

  ! [rejected]        master -> master (non-fast-forward)

what this means is that when we assumed they had:

...--F--G--H   <-- master

we were just plain wrong. They had something like this instead:

             J   <-- master
            /
...--F--G--H

in their repository. We sent them our commit I, which says that commit H is its parent. They said OK—that's fine—and they now have:

             J   <-- master
            /
...--F--G--H
            \
             I

But then we asked them to please, if it was OK, move their master to identify commit I. If they were to do that, they would lose their commit J!

Remember, Git—any Git—works by starting with the branch name to find the last commit, then working backwards. If we have them set their master to remember commit I, and they work backwards from there, they will have commit I preceded by commit H, and then from there backwards as before. But commit J isn't backwards from H. It's forwards.

How to proceed from here

What we need to do, then, if we don't want to have them throw away commit J after all, is get their commit from them. We can have our Git call up their Git and get commit J, and put it into our Git. This gives us:

             J   <-- origin/master
            /
...--F--G--H
            \
             I   <-- master

in our repository. Our commit I still exists, and our master still remembers our commit I.

What we need now is to build some new commit—let's call it commit K, though in reality it will have some big ugly random-looking hash ID—that builds on their commit J. We'll take our snapshot, in our commit I, and modify it into a snapshot that starts with their snapshot in J but takes whatever we changed when we went from H to I.

You have one extra complication, of course. You have a configuration file in the way, that you decided should be stored in Git (a mistake, but one you are stuck with). You may need to move yours out of the way—perhaps out of your own Git repository entirely—and put the production version in place, and clear the skip-worktree bit.

Once you have done all of that, you can have any commit—including H, I, and/or J—come out of your Git repository into your work-tree and into Git's index. Your configuration is safely out of your work-tree entirely; you're working with the production version. You can now copy commit I to a new commit K that follows J. There are multiple ways to do that but perhaps the easiest is:

git checkout -b update-prod origin/master    # make a branch to use commit `J`

which gives you something that we can draw like this:

             J   <-- update-prod (HEAD), origin/master
            /
...--F--G--H
            \
             I   <-- master

It's now easy to use git cherry-pick to copy commit I. Normally I would call the copy I', but I've been using the name K, so let's write the command and draw the result:

git cherry-pick master

             J   <-- origin/master
            / \
...--F--G--H   K   <-- update-prod (HEAD)
            \
             I   <-- master

You can now rename your old master to old-master and your current update-prod to master:

git branch -m master old-master
git branch -m update-prod master

which gives you:

             J   <-- origin/master
            / \
...--F--G--H   K   <-- master (HEAD)
            \
             I   <-- old-master

(You probably don't even need the name old-master any more, but it does not hurt to leave it around for a while.) Now you can set the skip-worktree bit on the configuration file again, copy the development configuration back into place, and test it as much as necessary.

When your testing is good, you can run:

git push origin master

This time, your Git calls up their Git, offers your new commit K, and then asks them to set their master to point to the now-shared commit K. Their Git no longer has a reason to reject the request. On their Git, then, this will produce:

...--F--G--H--J--K   <-- master

Commit I is not visible in their Git. It may be in there, because you sent it earlier, but nobody can find it because the way we—including Git—find commits is to start with the branch names and use those to find the last commits, and then work backwards, and there is no name in their Git by which to find commit I.

Once you delete your own name old-master, you no longer have a convenient way to find commit I either, and it will be as though it never existed. We'll have no reason to draw the commits with a kink in them, as your own Git will now have:

...--F--G--H--J--K   <-- master (HEAD), origin/master

Your Git will update your own origin/master—your Git's memory of their Git's master—as soon as they accept your git push request.