CODEWITHSUNDEEP
gitgit-submodules
evgeniuz
evgeniuz

Reputation: 2769

Restore git submodules from .gitmodules

I have a folder, which was a git repo. It contains some files and .gitmodules file. Now, when I do git init and then git submodule init, the latter command output is nothing. How can I help git to see submodules, defined in .gitmodules file without running git submodule add by hand again?

Update: this is my .gitmodules file:

[submodule "vim-pathogen"]
    path = vim-pathogen
    url = git://github.com/tpope/vim-pathogen.git
[submodule "bundle/python-mode"]
    path = bundle/python-mode
    url = git://github.com/klen/python-mode.git
[submodule "bundle/vim-fugitive"]
    path = bundle/vim-fugitive
    url = git://github.com/tpope/vim-fugitive.git
[submodule "bundle/ctrlp.vim"]
    path = bundle/ctrlp.vim
    url = git://github.com/kien/ctrlp.vim.git
[submodule "bundle/vim-tomorrow-theme"]
    path = bundle/vim-tomorrow-theme
    url = git://github.com/chriskempson/vim-tomorrow-theme.git

and here is listing of this dir:

drwxr-xr-x  4 evgeniuz 100 4096 июня  29 12:06 .
drwx------ 60 evgeniuz 100 4096 июня  29 11:43 ..
drwxr-xr-x  2 evgeniuz 100 4096 июня  29 10:03 autoload
drwxr-xr-x  7 evgeniuz 100 4096 июня  29 12:13 .git
-rw-r--r--  1 evgeniuz 100  542 июня  29 11:45 .gitmodules
-rw-r--r--  1 evgeniuz 100  243 июня  29 11:18 .vimrc

so, definitely, it is in top level. the git directory is not changed, only git init is done

Upvotes: 106

Views: 52750

Answers (9)

apfreelance
apfreelance

Reputation: 34

Just in case, someone doesnt want to use bash-script, here's a pythonscript that takes in a (from other repo copied) '.gitmodules' and from there creates a txtfile wich contains a list of git submodule add <path>;

with open("gitmodulesoutput.txt", "w") as output_file:
    with open(".gitmodules") as file:
        for line in file:
            line = line.strip()
            line = line.replace("url = ", "")
            line = "git submodule add " + line + ";"
            if ".git" in line:
                output_file.write(line + "\n")

then you only have to copy the newly created filecontent into a terminal and wait for git to do the magic.

Upvotes: 0

Mark Longair
Mark Longair

Reputation: 467261

git submodule init only considers submodules that already are in the index (i.e. "staged") for initialization. I would write a short script that parses .gitmodules, and for each url and path pair runs:

git submodule add <url> <path>

For example, you could use the following script:

#!/bin/sh

set -e

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key local_path
    do
        url_key=$(echo $path_key | sed 's/\.path/.url/')
        url=$(git config -f .gitmodules --get "$url_key")
        git submodule add $url $local_path
    done

This is based on how the git-submodule.sh script itself parses the .gitmodules file.

Upvotes: 137

Petr Plenkov
Petr Plenkov

Reputation: 109

I'm so surprised to see so many scripts for a very simple action and unfortunately I haven't found any tool or ready for use command. The only software so far which allows to restore it - is Gitkraken. Check here: https://help.gitkraken.com/gitkraken-client/submodules/

Upvotes: 2

nhooyr
nhooyr

Reputation: 1204

For zsh users, try my function which has DRY_RUN=1 support to see what commands will be ran and only uses git to parse the file instead of sed.

gsub_file() {(
  set -eu

  cd "$(git rev-parse --show-toplevel)"

  submodule_paths=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "path" | awk '{ print $2 }')}"
  )
  submodule_urls=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "url" | awk '{ print $2 }')}"
  )
  submodule_branches=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "branch" | awk '{ print $2 }')}"
  )

  sh_c() {
    echo + "$*"
    if [ "${DRY_RUN-}" ]; then
      return
    fi
    eval "$@"
  }

  for (( i=1; i <= ${#submodule_paths[@]}; i++ )) do
    p="${submodule_paths[$i]}"
    if [ -d "$p" ]; then
      continue
    fi

    url="${submodule_urls[$i]}"
    unset b
    if [ "${submodule_branches[$i]-}" ]; then
      b="-b ${submodule_branches[$i]}" 
    fi
    sh_c git submodule add "${b-}" "$url" "$p"
  done
)}

Upvotes: 0

bokov
bokov

Reputation: 3534

An updated version of the script by @mark-longair . This one also supports branches, handles the case where some of the submodules already exist in .git/config, and when necessary backs up existing directories of the same name as the submodule paths.

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        url_key=$(echo $path_key | sed 's/\.path/.url/');
        branch_key=$(echo $path_key | sed 's/\.path/.branch/');
        # If the url_key doesn't yet exist then backup up the existing
        # directory if necessary and add the submodule
        if [ ! $(git config --get "$url_key") ]; then
            if [ -d "$path" ] && [ ! $(git config --get "$url_key") ]; then
                mv "$path" "$path""_backup_""$(date +'%Y%m%d%H%M%S')";
            fi;
            url=$(git config -f .gitmodules --get "$url_key");
            # If a branch is specified then use that one, otherwise
            # default to master
            branch=$(git config -f .gitmodules --get "$branch_key");
            if [ ! "$branch" ]; then branch="master"; fi;
            git submodule add -f -b "$branch" "$url" "$path";
        fi;
    done;

# In case the submodule exists in .git/config but the url is out of date

git submodule sync;

# Now actually pull all the modules. I used to use this...
#
# git submodule update --init --remote --force --recursive
# ...but the submodules would check out in detached HEAD state and I 
# didn't like that, so now I do this...

git submodule foreach --recursive 'git checkout $(git config -f $toplevel/.gitmodules submodule.$name.branch || echo master)';

Upvotes: 6

mauron85
mauron85

Reputation: 1544

Extending excellent @Mark Longair's answer to add submodule respecting branch and repo name.

#!/bin/sh

set -e

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        name=$(echo $path_key | sed 's/\submodule\.\(.*\)\.path/\1/')
        url_key=$(echo $path_key | sed 's/\.path/.url/')
        branch_key=$(echo $path_key | sed 's/\.path/.branch/')
        url=$(git config -f .gitmodules --get "$url_key")
        branch=$(git config -f .gitmodules --get "$branch_key" || echo "master")
        git submodule add -b $branch --name $name $url $path || continue
    done

Upvotes: 11

alexgirao
alexgirao

Reputation: 895

I know its been a while, but I want to share this version that calls git config only once, doesn't requires a script and also handles branches:

git config -f .gitmodules --get-regexp '^submodule\.' | perl -lane'
$conf{$F[0]} = $F[1]}{
@mods = map {s,\.path$,,; $_} grep {/\.path$/} keys(%conf);
sub expand{$i = shift; map {$conf{$i . $_}} qw(.path .url .branch)}
for $i (@mods){
    ($path, $url, $branch) = expand($i);
    print(qq{rm -rf $path});
    print(qq{git submodule add -b $branch $url $path});
}
'

The only side effect is the output of the commands, nothing gets executed, so you can audit before committing to them.

This works with a simple copy and paste at the console, but should be trivial to put in a shell script.

example output:

rm -rf third-party/dht
git submodule add -b post-0.25-transmission https://github.com/transmission/dht third-party/dht
rm -rf third-party/libutp
git submodule add -b post-3.3-transmission https://github.com/transmission/libutp third-party/libutp
rm -rf third-party/libb64
git submodule add -b post-1.2.1-transmission https://github.com/transmission/libb64 third-party/libb64
rm -rf third-party/libnatpmp
git submodule add -b post-20151025-transmission https://github.com/transmission/libnatpmp third-party/libnatpmp
rm -rf third-party/miniupnpc
git submodule add -b post-2.0.20170509-transmission https://github.com/transmission/miniupnpc third-party/miniupnpc
rm -rf third-party/libevent
git submodule add -b post-2.0.22-transmission https://github.com/transmission/libevent third-party/libevent

Upvotes: 1

Henry
Henry

Reputation: 7891

I had a similar issue. git submodule init was failing silently.

When I did:

git submodule add <url> <path>

I got:

The following path is ignored by one of your .gitignore files: ...

I'm thinking that .gitignore(d) paths might be the cause.

Upvotes: 3

Kevin C.
Kevin C.

Reputation: 2527

Expanding on @Mark Longair's answer, I wrote a bash script to automate steps 2 & 3 of the following process:

  1. Clone a 'boilerplate' repo to begin a new project
  2. Remove the .git folder and re-initialize as a new repo
  3. Re-initialize the submodules, prompting for input before deleting folders
#!/bin/bash

set -e
rm -rf .git
git init

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' > tempfile

while read -u 3 path_key path
do
    url_key=$(echo $path_key | sed 's/\.path/.url/')
    url=$(git config -f .gitmodules --get "$url_key")

    read -p "Are you sure you want to delete $path and re-initialize as a new submodule? " yn
    case $yn in
        [Yy]* ) rm -rf $path; git submodule add $url $path; echo "$path has been initialized";;
        [Nn]* ) exit;;
        * ) echo "Please answer yes or no.";;
    esac

done 3<tempfile

rm tempfile

Note: the submodules will be checked out at the tip of their master branch instead of the same commit as the boilerplate repo, so you'll need to do that manually.

Piping the output from git config into the read loop was causing problems with the prompt for input, so it outputs it to a temp file instead. Any improvements on my first bash script would be very welcome :)

Big thanks to Mark, https://stackoverflow.com/a/226724/193494, bash: nested interactive read within a loop that's also using read, and tnettenba @ chat.freenode.net for helping me arrive at this solution!

Upvotes: 17

Related Questions