CODEWITHSUNDEEP
dockercontainersibm-cloudpipelinedevops
mc9
mc9

Reputation: 6349

Bluemix Dev Ops: Building a project with private git submodules

I would like to know how to build a project with private git submodules using IBM Bluemix Dev Ops Services.

In my pipelines, I have a 'Build' job with the type 'Shell Script':

#!/bin/bash
git submodule init
git submodule update --recursive

But my submodules include a number of private repositories, and I get:

Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

In my local machine, I am able to run those commands because I have access and I am using my key. What can I do to make it work here? I do not wish to commit my private key into git.

The repo for the app I am deploying is hosted on GitHub. And the private submodules are hosted on BitBucket.

Update

I tried to use my private key in the build console, but it did not work:

echo "... my private key ..." >> ~/.ssh/throwaway_key
chmod 400 ~/.ssh/throwaway_key
ssh-agent bash -c 'ssh-add ~/.ssh/throwaway_key; git submodule update --recursive'

Is it not working because I am inside a docker container? Do I have to update /etc/ssh/ssh_config? I don't have access to this inside the container that this job runs in.

Update 2

I also tried without success:

echo "Host            bitbucket.org
    Hostname        bitbucket.org
    IdentityFile    ~/.ssh/throwaway_key
    IdentitiesOnly yes" >> ~/.ssh/config

Upvotes: 3

Views: 358

Answers (3)

Bohr
Bohr

Reputation: 1882

Here is a simple solution:

  1. Change the Build stage type from "Simple" to "Shell Script".

  2. Add this to script:

     git submodule init
 git config submodule.foo.url https://$USERNAME:[email protected]/foo.git
 git submodule update --recursive

  3. Add USERNAME and PASSWORD in the Environment Properties.

credit: https://stackoverflow.com/a/7714592/1335313

Problem with the above approach is that you need to work in the delivery pipeline to setup for each submodule. It can create a problem when the submodules changed.

Another potential solution is to setup git credential store in the pipeline:

git config credential.helper store
echo "https://LOGIN:${PAT}@github.com/path/to/submodule.git" > ~/.git-credentials

Credit: https://stackoverflow.com/a/63939958/1335313

Read more: https://www.shellhacks.com/git-config-username-password-store-credentials/

Upvotes: 0

xverges
xverges

Reputation: 4718

This worked for me, using a github deploy key for the repo where my submodule resides

    #!/bin/bash

    # Build the private key file from the secret environment setting $id_rsa_abacus_pylib
    echo "-----BEGIN OPENSSH PRIVATE KEY-----" > /home/pipeline/.ssh/id_rsa_abacus_pylib
    (echo $id_rsa_abacus_pylib | tr ' ' '\n' | grep -v -e "----" | grep -v -e OPENSSH | grep -v -e PRIVATE) >> /home/pipeline/.ssh/id_rsa_abacus_pylib
    echo "-----END OPENSSH PRIVATE KEY-----" >> /home/pipeline/.ssh/id_rsa_abacus_pylib
    chmod 400 /home/pipeline/.ssh/id_rsa_abacus_pylib
    #cat /home/pipeline/.ssh/id_rsa_abacus_pylib

    # Replace the SSH command used by git
    echo 'ssh -vvv -i ~/.ssh/id_rsa_abacus_pylib -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $*' > ssh
    chmod +x ssh

    GIT_SSH="./ssh" git submodule init
    GIT_SSH="./ssh" git submodule update --recursive

In my initial attempts, the operation did not work because I was failing to properly move the SSH key from an environment variable to a file.

It takes advantage from option 3 in https://superuser.com/questions/232373/how-to-tell-git-which-private-key-to-use/868699#868699

Upvotes: 0

Holly Cummins
Holly Cummins

Reputation: 11492

I have a similar setup. I define a Checkout job, whose job it is to re-extract the source, explicitly passing a password in the clone URL. Once that's done, the submodule update works fine. This is the script:

#!/bin/bash
git clone --recursive https://myname:[email protected]/git/myname/my-project
cd my-project
git submodule update --remote

PASSWORD is defined as a Secure property on the Environment Properties tab. It's a bit clunky and non-DRY, but it enabled the behaviour I wanted.

I use the Checkout job as an input to the Build job (I probably could have done it as one big job, but I wanted to be able to visually distinguish failures in checkout and build.)

Upvotes: 1

Related Questions