CODEWITHSUNDEEP
gitbashgitlab
Mamun Sardar
Mamun Sardar

Reputation: 2729

bash script to clone/pull bulk git repo

I've several(22) private repositories on gitlab with https protocol and I want to checkout all repositories providing username and password once from bash script. But can't seem to make it work. Here's how far I got: (any edit/optimization is much appreciated):

#!/bin/bash

read -p "gitlab username: " user
read -sp "gitlab password: " pass
echo
read -p "branch to checkout: " branch
echo

repo[0]='.' #dir
repo[1]='repo.myhost.com/project/app.git' #repo url

repo[2]='plugins'
repo[3]='repo.myhost.com/project/app-core.git'

repo[4]='plugins'
repo[5]='repo.myhost.com/project/plugin1.git'

repo[6]='plugins'
repo[7]='repo.myhost.com/project/plugin2.git'

repo[8]='api-plugins'
repo[9]='repo.myhost.com/project/api-plugin1.git'

repo[10]='api-plugins'
repo[11]='repo.myhost.com/project/api-plugin2.git'

# will add more repo

total=${#repo[*]}
echo "checking out repositories..."
mkdir -p plugins
mkdir -p api-plugins

for (( i=0; i<${total}; i+=2 ));
do
    dir=${repo[$i]}
    trepo="https://$user:$pass@${repo[$i+1]}"
    echo "checking out ${repo[$i+1]} to directory $dir"...
    if cd $dir; then git pull; else git clone -b branch --single-branch $trepo $dir; fi
    echo
done

Edit: My git pull doesn't provide any password, don't know how to do that. Also my password contains @ in it.

The outcome of this script: asking for username/pass again

mamun@linux ~/dev/projects $ ./checkout.sh
git username: myuser
git password: 
project branch: master

checking out repositories...
checking out repo.myhost.com/project/app.git to directory ....
Username for 'https://repo.myhost.com': ^C

Upvotes: 1

Views: 3009

Answers (2)

VonC
VonC

Reputation: 1323463

As mentioned, a password with @ must be percent encoded (as I mention here).
In pure bash, see this function for instance.

Try in your same bash those commands:

echo git ls-remote http://${user}:${pass}@repo.myhost.com/project/app.git
git ls-remote http://${user}:${pass}@repo.myhost.com/project/app.git

The ls-remote will query the remote repo branches, without pulling or cloning: this is just for testing.

The OP Mamun Sardar confirms in the comments the encoding issue and adds:

Also I made a mistake should use: git clone -b $branch instead of git clone -b branch

I agree and would recommend:

Upvotes: 0

zigarn
zigarn

Reputation: 11595

One easy solution would be to use SSH protocol with SSH key. If the key have a passphrase, you can use ssh-agent.

If you're stuck with HTTP(S), git provides a mechanism to give the username and password without putting them in the URL: GIT_ASKPASS

GIT_ASKPASS is an env-var that points to a script that take one argument: the prompt ("Username for ..." or "Password for ...") and outputs the prompted value.
So a solution is:

GIT_ASKPASS=$(mktemp)
cat <<EOF > ${GIT_ASKPASS}
#!/bin/sh
case "\$1" in
    Username*) echo '${user}' ;;
    Password*) echo '${pass}' ;;
esac
EOF
chmod +x ${GIT_ASKPASS}

# Put here all your git calls that need credentials

rm -f ${GIT_ASKPASS}

Upvotes: 1

Related Questions