Google App Engine deployment with private git repo in package.json

Question

I have a dependency to a private Bitbucket repo in my package.json

{
   "my-dependency": "git+ssh://git@bitbucket.org/something/my-dependency.git"
}

I followed the instructions given in [1] and [2] and created an SSH key that I encrypted with kms.

I've created a custom cloudbuild.yaml as follows:

# Decrypt the file containing the key
steps:
  - name: 'gcr.io/cloud-builders/gcloud'
    args:
      - kms
      - decrypt
      - --ciphertext-file=bitbucket_rsa.enc
      - --plaintext-file=/root/.ssh/id_rsa
      - --location=global
      - --keyring=default
      - --key=bitbucket-key
    volumes:
      - name: 'ssh'
        path: /root/.ssh

  # Set up git with key and domain
  - name: 'gcr.io/cloud-builders/git'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        chmod 600 /root/.ssh/id_rsa
        cat <<EOF >/root/.ssh/config
        Hostname bitbucket.org
        IdentityFile /root/.ssh/id_rsa
        EOF
        mv known_hosts /root/.ssh/known_hosts
    volumes:
      - name: 'ssh'
        path: /root/.ssh

  # Install
  - name: 'gcr.io/cloud-builders/yarn'
    args: ['install']
    volumes:
      - name: 'ssh'
        path: /root/.ssh

  # Build
  - name: "gcr.io/cloud-builders/yarn"
    args: ["build"]
    volumes:
      - name: 'ssh'
        path: /root/.ssh

  # Deploy
  - name: "gcr.io/cloud-builders/gcloud"
    args: ["app", "deploy", "my-service.yaml"]
    volumes:
      - name: 'ssh'
        path: /root/.ssh

When I run it via gcloud builds submit --config=cloudbuild.yaml steps #0 to #3 run through fine, but step #4 fails because app deploy triggers another yarn install which does not have access to the SSH key defined in steps #0 and #1:

Step #4: INFO     rm_node_modules took 0 seconds
Step #4: INFO     starting: yarn_install
Step #4: INFO     yarn_install yarn install
Step #4: INFO     `yarn_install` stdout:
Step #4: yarn install v1.9.4
Step #4: [1/5] Validating package.json...
Step #4: [2/5] Resolving packages...
Step #4: [3/5] Fetching packages...
Step #4: info Visit https://yarnpkg.com/en/docs/cli/install for     documentation about this command.
Step #4:
Step #4: INFO     `yarn_install` had stderr output:
Step #4: error Command failed.
Step #4: Exit code: 128
Step #4: Command: git
Step #4: Arguments: ls-remote --tags --heads     ssh://git@bitbucket.org/something/my-dependency.git
Step #4: Directory: /workspace
Step #4: Output:
Step #4: Host key verification failed.
Step #4: fatal: Could not read from remote repository.
Step #4:
Step #4: Please make sure you have the correct access rights
Step #4: and the repository exists.
Step #4:
Step #4: ERROR    error: `yarn_install` returned code: 1
Step #4: INFO     yarn_install took 11 seconds
Step #4: INFO     build process for FTL image took 11 seconds
Step #4: INFO     full build took 11 seconds
Step #4: ERROR    `yarn_install` had stderr output:
Step #4: error Command failed.

Thank you for your help!

References:

[1] https://cloud.google.com/cloud-build/docs/access-private-github-repos

[2] Link private repository in packages.json in app deployed to gcloud

Answer 1

So apparently it is not possible to provide an SSH key for the gcloud app deploy step. Thus using

{
   "my-dependency": "git+ssh://git@bitbucket.org/something/my-dependency.git"
}

will not work!

The workaround (as mentioned by @JKleinne in the linked thread) is to clone the repo and install it from a local folder:

{
  "my-dependency": "lib/my-dependency"
}

I wrote a small bash script that checks if the repo can be accessed and clones/pulls if it can:

GIT_PROJECT=$1
GIT_REPO=$2
NAME=${GIT_REPO}
REMOTE="git@bitbucket.org:${GIT_PROJECT}/${GIT_REPO}.git"

if [[ ! -d ./lib ]]
then
    mkdir -p ./lib
fi

## Test if git repo is accessible
if ! git ls-remote --exit-code -h ${REMOTE}; then
    echo "Unable to access git repo, skipping"
    exit 0
fi

## Clone or pull
if [[ ! -d ./lib/${NAME} ]]
then
    git clone ${REMOTE} ./lib/${NAME}
else
    git -C ./lib/${NAME} pull
fi

I then use it in the preinstall script:

"preinstall": "./get-internal-package.sh something my-dependency",

Answer 2

Whenever there's a Host Key verification failed error, it is most likely related to your known_hosts file, it means that the host key of bitbucket.org is not in your known_hosts file and so the client does not have a way to verify it. Try running ssh-keyscan -t rsa bitbucket.org > known_hosts and then cat known_hosts and see if the bitbucket.org host key is present.

If the answer is no and the output is empty, it could be a network problem interfering with the process. Follow the accepted answer in this thread to troubleshoot it.

If the answer is yes, perfect there is only one step missing and that is to configure your SSH key into Bitbucket. Go into bitbucket.org under "Bitbucket Settings" and then click "SSH Keys" in one of the options. Add a key (label it whichever name you decide) and paste the output you get from running cat ~/.ssh/id_rsa.pub into the key section.

Essentially, in one of the threads you linked I'm cloning the private repository in the same folder as the root of my application. That way, in the package.json I can simply add this line dependencies: {“circular-structure-stringify”: “./circular-structure-stringify”} which will allow me to require() it like any npm package.

I realize I should've kept the Github link open but nevertheless, I am working on another example which I will be posting in the comments section soon. Meanwhile, let me know on the outcome.