CODEWITHSUNDEEP
gitgit-submodules
David Eyk
David Eyk

Reputation: 12521

Git will not init/sync/update new submodules

Here's part of the contents of my .gitmodules file:

[submodule "src/static_management"]
        path = src/static_management
        url = git://github.com/eykd/django-static-management.git
[submodule "external/pyfacebook"]
        path = external/pyfacebook
        url = http://github.com/sciyoshi/pyfacebook.git

However, .git/config only contains the first:

[submodule "src/static_management"]
        url = git://github.com/eykd/django-static-management.git

The second submodule (external/pyfacebook) was added by another developer in a feature branch. I've inherited the development now, and have checked out the feature branch. However, Git will not pull the submodule for me. I've tried:

in various combinations, but git simply will not update .git/config based on the new contents of .gitmodules, nor will it create the external/pyfacebook folder and pull the submodule's contents.

What am I missing? Is manual intervention (adding a submodule entry by hand to .git/config) truly required, and why?

Edit: Manual intervention does not work. Manually adding the new submodule entry to .git/config doesn't do a thing. The new submodule is ignored.

Upvotes: 218

Views: 244105

Answers (24)

Tatsh
Tatsh

Reputation: 3680

Special scenario if you grabbed a source tarball/zip with no .git but still has .gitmodules at the root. All submodule URLs have to still be valid. If the loop fails, clean up .git/config of submodule entries and rm -rf .git/modules/*.

Always make backups!

Re-construct the repository:

git init
git add .
git commit -m 'First commit'

Fetch all of the submodules:

while IFS=$' ' read -r path url; do git rm -rf "$path" &>/dev/null; rm -rf "$path"; mkdir -p "$(dirname "$path")" && git submodule add "$url" "$path"; done < <(grep -A 1 'path =' .gitmodules | perl -lpe 's/^\s+|\s+$//g' | sed ':a;N;$!ba;s/\nurl =/ /g' | grep -Fv -- -- | sed -re 's/^path = //')

Or more nicely:

while IFS=$' ' read -r path url; do
  git rm -rf "$path" &>/dev/null
  rm -rf "$path"
  mkdir -p "$(dirname "$path")"
  git submodule add "$url" "$path"
done < <(
  grep -A 1 'path =' .gitmodules |
  perl -lpe 's/^\s+|\s+$//g' |
  sed ':a;N;$!ba;s/\nurl =/ /g' |
  grep -Fv -- -- |
  sed -re 's/^path = //'
)

Input pipeline of the loop (the < <() part):

  • grep -A 1 'path =' .gitmodules - Get the path = line and the url = line that comes after in .gitmodules
  • perl -lpe 's/^\s+|\s+$//g' Trim the lines
  • sed ':a;N;$!ba;s/\nurl =/ /g' - merge the url = line with the prior line
  • grep -Fv -- -- - remove the ^--$ lines caused by use of grep -A
  • sed -re 's/^path = //' - remove the path = prefix of every line

Body explained:

  • Use git rm to remove the module storage path just in case
  • Delete the path by normal means
  • Create the parent path (dirname gets the parent dir path)
  • Add the submodule

Upvotes: 1

Dave James Miller
Dave James Miller

Reputation: 5298

I had this same problem - it turned out that the .gitmodules file was committed, but the actual submodule commit (i.e. the record of the submodule's commit ID) was not.

Adding it manually seemed to do the trick - e.g.:

git submodule add http://github.com/sciyoshi/pyfacebook.git external/pyfacebook

(Even without removing anything from .git/config or .gitmodules.)

Then commit it to record the ID properly.

Adding some further comments to this working answer: If the git submodule init or git submodule update does not work, then as described above git submodule add <url> should do the trick. One can cross check this by

 git config --list

and one should get an entry of the submodule you want to pull in the result of the git config --list command. If there is an entry of your submodule in the config result, then now the usual git submodule update --init should pull your submodule. To test this step, you can manually rename the submodule and then updating the submodule.

 mv yourmodulename yourmodulename-temp
 git submodule update --init

To find out if you have local changes in the submodule, it can be seen via git status -u ( if you want to see changes in the submodule ) or git status --ignore-submodules ( if you dont want to see the changes in the submodule ).

Upvotes: 182

Liplum
Liplum

Reputation: 19

According to the book of git, if you want to "pull" the default branch of a submodule into the local repo that you currently working on, try git submodule update --remote.

To learn about how to track another branch, please carefully check the document mentioned above.

There is an easier way to do this as well, if you prefer to not manually fetch > and merge in the subdirectory. If you run git submodule update --remote, Git will go into your submodules and fetch and update for you.

$ git submodule update --remote DbConnector
remote: Counting objects: 4, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 4 (delta 2), reused 4 (delta 2)
Unpacking objects: 100% (4/4), done.
From https://github.com/chaconinc/DbConnector
   3f19983..d0354fc  master     -> origin/master
Submodule path 'DbConnector': checked out 'd0354fc054692d3906c85c3af05ddce39a1c0644'

Upvotes: 1

thzu
thzu

Reputation: 63

I had the same problem and in my case the solution was very easy:

  1. delete empty folder (name of submodule)
  2. run git submodule update

Upvotes: 1

DZet
DZet

Reputation: 579

This means the submodules haven’t been set up correctly and a git submodule add command will have to be executed. A detailed explanation of how submodules work:

If submodules were not added correctly:

If not committed correctly, and the submodules and their corresponding folders are not already recorded in the index, then first every submodule in the .gitmodules file needs to be individually added via git submodule add before one can proceed.

  • Note: as mentioned here, Git does not currently have a command to do this at once for multiple submodules present in a .gitmodules file.

What does git submodule add do?

When a repo with submodules has been set up correctly and someone has already performed a git submodule add command, the command has done the following things:

  1. create a folder for the submodule to reside in if it doesn’t already exist
  2. clone the project as a submodule
  3. set the submodule parameters in the shareable .gitmodules file
  4. set the submodule parameters in your private .git/config file
  5. create a .git folder for the submodule in your superproject’s private .git/modules folder
  6. and also: record in your superproject’s index for each submodule in which folder it will reside, and in what state it should be (the hash commit code of the submodule.)

Point 3 and 6 are relevant when you clone the superproject, and point 6 indicates whether a git submodule add has been performed correctly and was committed. You can check point 6 by seeing whether the folders in which the submodules should reside are already there and empty (Note: this does not require a .keep or .gitignore mechanism to be in place, since this is part of a Git mechanism.) After cloning the superproject you can also perform a git submodule to see which submodules are expected.

You can then proceed by doing:

  • git submodule

    will show the submodules present in the tree and their corresponding commit hash code, can serve as an initial check to see which submodules are expected

  • git submodule init

    copies the submodules parameters from the repo’s .gitmodules file to your private .git/config file (point 4)

  • git submodule update

    clones the submodules to a commit determined by the superproject and creates the submodules' .git folders under your superproject's .git/modules/ folder (points 2 and 5)

  • git submodule update --remote

    same as update but sets the submodule to the latest commit on the branch available by the remote repo, similar as going in each submodule’s folder afterwards and do a git pull

  • or --->: git submodule update --init --remote

    which is all of the above combined.

Alternatively, when the repo is set up correctly, you can also do a git clone with the --recursive flag to include the submodules with the init and update performed on them automatically.

Upvotes: 12

Black
Black

Reputation: 20212

Please check your submodules directory.

If there is only a .git file in it, then delete it.

Now execute git submodule update --remote --init

Upvotes: -1

Nishant Chauhan
Nishant Chauhan

Reputation: 744

Below sync command resolved the issue :

git submodule sync

Upvotes: 2

Anne Kariny Silva Freitas
Anne Kariny Silva Freitas

Reputation: 85

Just sharing what worked for me: 

git clone --recurse-submodules <repository path>

This clones the remote repository already including the submodules. This means you won't need to run git submodule update or init after cloning.

Upvotes: 1

Venryx
Venryx

Reputation: 17979

The problem for me is that the repo's previous developer had committed the submodules/thing folder as just a regular folder, meaning when I tried to run git submodule add ..., it would fail with: 'submodules/thing' already exists in the index, yet trying to update the submodule would also fail because it saw that the path did not contain a submodule.

To fix, I had to delete the submodules/thing folder, commit the deletion, then run the git submodule add command to add it back correctly:

git submodule add --force --name thing https://github.com/person/thing.git submodules/thing

Upvotes: 6

masterop
masterop

Reputation: 184

I had the same problem but none of the solutions above helped. The entries in the .gitmodules and in .git/config were right but the command git submodules update --init --recursive was doing nothing. I also removed the submodule directory and did run git submodules update --init --recursive and got the submodule directory back but with exactly the same commit as before.

I found the answer on this page. The command is:git submodule update --remote

Upvotes: 13

farinspace
farinspace

Reputation: 8791

Thinking that manually setting up .gitmodules is enough is WRONG

My local git version 2.22.0 as of this writing.

So I came to this thread wondering why wasn't git submodule init working; I setup the .gitmodules file and proceeded to do git submodule init ...

IMPORTANT

  1. git submodule add company/project.git includes/project is required (when adding the module for the first time), this will:

    • add config to .git/config
    • update the .gitmodules file
    • track the submodule location (includes/project in this example).

  2. you must then git commit after you have added the submodule, this will commit .gitmodules and the tracked submodule location.

When the project is cloned again, it will have the .gitmodules and the empty submodules directory (e.g. includes/project in this example). At this point .git/config does not have submodule config yet, until git submodule init is run, and remember this only works because .gitmodules AND includes/project are tracked in the main git repo.

Also for reference see:

Upvotes: 26

Masoud Darvishian
Masoud Darvishian

Reputation: 3964

  • Remove the submodule from your .git/config
  • Run git submodule init command
  • Go to your submodule directory and run git pull origin master

It should works now

Upvotes: -1

Marc
Marc

Reputation: 106

For the record:
I created the same issue by adding an empty repository as submodule. In this case, there was no reference hash available for the submodule, leading to the error described by the original poster.

Force-adding the repository after having committed to it solved the issue (as in Arvids post)
git submodule add --force [email protected] destination

Upvotes: 4

Arvid
Arvid

Reputation: 1081

I had a similar problem with a submodule. It just didn't want to be cloned/pulled/updated/whatever.

When trying to re-add the submodule using git submodule add [email protected] destination I got the following output:

A git directory for 'destination' is found locally with remote(s):
  origin        [email protected]
If you want to reuse this local git directory instead of cloning again from
  [email protected]
use the '--force' option. If the local git directory is not the correct repo
or you are unsure what this means choose another name with the '--name' option.

So, I tried to enforce the add command:
git submodule add --force [email protected] destination

That worked in my case.

Upvotes: 3

Carlo Wood
Carlo Wood

Reputation: 6791

There seems to be a lot of confusion here (also) in the answers.

git submodule init is not intended to magically generate stuff in .git/config (from .gitmodules). It is intended to set up something in an entirely empty subdirectory after cloning the parent project, or pulling a commit that adds a previously non-existing submodule.

In other words, you follow a git clone of a project that has submodules (which you will know by the fact that the clone checked out a .gitmodules file) by a git submodule update --init --recursive.

You do not follow git submodule add ... with a git submodule init (or git submodule update --init), that isn't supposed to work. In fact, the add will already update the appropriate .git/config if things work.

EDIT

If a previously non-existing git submodule was added by someone else, and you do a git pull of that commit, then the directory of that submodule will be entirely empty (when you execute git submodule status the new submodule's hash should be visible but will have a - in front of it.) In this case you need to follow your git pull also with a git submodule update --init (plus --recursive when it's a submodule inside a submodule) in order to get the new, previously non-existing, submodule checked out; just like after an initial clone of a project with submodules (where obviously you didn't have those submodules before either).

Upvotes: 10

atahan
atahan

Reputation: 706

Deleting submodule dir and its contents ("external/pyfacebook" folder) if it exists before git submodule add ... might fix problems.

Upvotes: 1

Eric Jeker
Eric Jeker

Reputation: 546

I had the same problem today and figured out that because I typed git submodule init then I had those line in my .git/config:

[submodule]
   active = .

I removed that and typed:

git submodule update --init --remote

And everything was back to normal, my submodule updated in its subdirectory as usual.

Upvotes: 3

palik
palik

Reputation: 2853

git version 2.7.4. This command updates local code git submodule update --init --force --remote

Upvotes: 127

Alex Ivasyuv
Alex Ivasyuv

Reputation: 8814

Had the same issue, when git ignored init and update commands, and does nothing.

HOW TO FIX

  1. Your submodule folder should be committed into git repo
  2. It shouldn't be in .gitignore

If that requirements met, it will work. Otherwise, all commands will execute without any messages and result.

If you did all that, and it still doesn't work:

  1. Add submodule manually, e.g. git submodule add git@... path/to
  2. git submodule init
  3. git submodule update
  4. commit and push all files - .gitmodules and your module folder (note, that content of folder will not commit)
  5. drop your local git repo
  6. clone a new one
  7. ensure that .git/config doesn't have any submodules yet
  8. Now, git submodule init - and you will see a message that module registered
  9. git submodule update - will fetch module
  10. Now look at .git/config and you will find registered submodule

Upvotes: 26

levifig
levifig

Reputation: 510

Sort of magically, but today I ran git submodule init followed by git submodule sync followed by git submodule update and it started pulling my submodules... Magic? Perhaps! This is truly one of the most annoying experiences with Git…

Scratch that. I actually got it working by doing git submodule update --init --recursive. Hope this helps.

PS: Make sure you are in the root git directory, not the submodule's.

Upvotes: 5

Phil P
Phil P

Reputation: 833

When I saw this today, a developer had moved part of the tree into a new sub-directory and it looks as though his git client did not record the updated Subproject rules in the tree, instead they were just nuked, leaving .gitmodules referring both to stale locations and to subprojects which no longer existed in the current tree.

Adding the submodules back in, and comparing the commit shas of the submodule to those found in git show $breaking_commit_sha (search for lines matching regexp ^-Subproject) to adjust as needed fixed things.

Upvotes: 1

fridh
fridh

Reputation: 11

Same as you I found that git submodule sync does not do what you expect it to do. Only after doing an explicit git submodule add again does a submodule url change.

So, I put this script in ~/bin/git-submodule-sync.rb:

https://gist.github.com/frimik/5125436

And I also use the same logic on a few post-receive git deploy scripts.

All I need to do now is edit .gitmodules, then run this script and it finally works like I thought git submodule sync was supposed to.

Upvotes: 1

joseph.hainline
joseph.hainline

Reputation: 26608

I had the same problem.

.gitmodules had the submodule, but after a git submodule init command it wasn't in .git/config.

Turns out the developer who added the submodule also added the submodule directory to the .gitignore file. That doesn't work.

Upvotes: 4

dirkaholic
dirkaholic

Reputation: 350

According to the answer from Dave James Miller I can confirm that it worked for me. The important thing here was to commit the subprojects commit ID. Just to have the entry in .gitmodules was not enough.

Here is an appropriate commit:

https://github.com/dirkaholic/vagrant-php-dev-box/commit/d5f4c40bdbd80eefbb5ac6029823733f591435ae

Upvotes: 4

Related Questions