CODEWITHSUNDEEP
node.jsgoogle-app-enginenpmgoogle-cloud-platformgcloud
johnatann
johnatann

Reputation: 145

Link private repository in packages.json in app deployed to gcloud

I have a node.js app (my-app) that I'm setting Continuous Deployment for. The repository is hosted on Bitbucket and I'm creating CD with Bitbucket's Pipelines. The pipelines script deploys the app to Google Cloud Platform, to App Engine.

I am encountering a problem when I try to add a package (my-package) to my-app that is in another private repository on Bitbucket. Here is an part of my packages.json file:

"dependencies": {
    "my-package": "git+ssh://[email protected]:team-name/my-package.git",
}

With such linking I was able to:

  1. Add my PC's ssh key to repository to make sure npm install works locally
  2. Add pipelines ssh key to repository to make sure that npm install works inside pipelines environment

What I'm struggling with is the gcloud app deploy command inside the pipelines script. To my best knowledge, this command, when deploying a node.js app, runs npm install inside Google's environment. Such environment doesn't have access to my private repository with my-package and subsequently throws this error:

ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build ##### status: FAILURE.
(...)
INFO     gen_package_lock npm install --package-lock-only
INFO     `gen_package_lock` stdout:
INFO     `gen_package_lock` had stderr output:
npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://[email protected]:team-name/my-package.git
npm ERR! 
npm ERR! Host key verification failed.
npm ERR! fatal: Could not read from remote repository.
npm ERR! 
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR! 
npm ERR! exited with error code: 128

The obvious solution for me would be a possibility to generate an ssh key inside the Google's environment and granting access to the private repository.

Is it possible? If not - what other solutions could be applied here?

Upvotes: 0

Views: 2145

Answers (1)

JKleinne
JKleinne

Reputation: 1310

I was able to make it work and will happily share with you the steps that I’ve taken. I’ve posted a working example in my GitHub so you can clone it and try it out for yourself, do note that I will be deleting this after a week. I will be going through the whole process from scratch and I encourage you to do so, even if you already have your SSH keys set up.

  1. First and foremost, clone the GitHub repo and cd into it.
  2. Create a cloud KMS KeyRing and CryptoKey. Assuming you have the Cloud SDK installed, run gcloud kms keyrings create my-keyring --location=global to create a keyring. Next, create the CryptoKey by running gcloud kms keys create bit-key --location=global --keyring=my-keyring --purpose=encryption.
  3. Next, create an SSH key by running ssh-keygen -t rsa -b 4096 -C [email protected]. It will prompt you for a paraphrase and a file to save the key, just press each time.
  4. Configure your SSH key into Bitbucket. Go into bitbucket.org under "Bitbucket Settings" and then click "SSH Keys" in one of the options. Add a key (label it whichever name you decide) and paste the output you get from running cat ~/.ssh/id_rsa.pub into the key section.
  5. Take note of where your id_rsa file is located and get its path relative to your project folder. For example, because I did this reproduction in the Cloud Shell, my .ssh/id_rsa file’s path relative to my project folder is simply ../.ssh/id_rsa. Then encrypt the SSH key by running gcloud kms encrypt --plaintext-file=<RELATIVE_PATH_TO_.SSH/id_rsa> --ciphertext-file=./id_rsa.enc --location=global --keyring=my-keyring --key=bit-key. In my example, would simply be ../.ssh/id_rsa
  6. Grant your Cloud Build service account decrypt permission by first going into console.cloud.google.com/iam-admin and find the Cloud Build service account: its name is similar to @cloudbuild.gserviceaccount.com and grant it the App Engine Admin role (this is so that we are able to run gcloud app deploy from Cloud Build later on) then copy the email address of that @cloudbuild.gserviceaccount.com service account and go to console.cloud.google.com/security/kms and select the checkbox of ‘my-keyring’. On the panel to the right, you will see the permissions and there add a new member and paste the service account you just copied. Grant it the Cloud KMS > Cloud KMS CryptoKey Decrypter role.
  7. Create or update the known_hosts file by running ssh-keyscan -t rsa bitbucket.org > known_hosts
  8. Run gcloud builds submit --config=cloudbuild.yaml

As you can see, I have a private repository hosted on bitbucket called circular-structure-stringify. From the cloudbuild.yaml, you will see that the SSH key is first decrypted into a plaintext located in /root/.ssh/id_rsa, which is then used in the next step to set up the key with Bitbucket.

Next, we clone the private repository from Bitbucket into my container, followed by an npm install and gcloud app deploy. As you will notice, the dependency now lives in the same folder as our application, which is the same folder where the package.json file is located.

Finally, in your package.json, add the dependency in the dependencies property like such: dependencies: {“circular-structure-stringify”: “./circular-structure-stringify”}. Subsequently, you can import the module like you would in any npm packages: const CircularStructureStringify = require(‘circular-structure-stringify’) like you can see in the /routes/index.js file.

Previous submission (summary of steps detailed above)

To answer your question, yes, it is possible. You will have to use Cloud KMS 1 to interact with a private Bitbucket repository. There exists a documentation explaining the steps needed to access a private Github repository [2], but it must be adjusted slightly to make it work with Bitbucket.

Furthermore, when generating an SSH key, make sure to provide the -C “[email protected]” is specified. From past experiences, I’ve had issues with Bitbucket specifically if the key didn’t have this set upon creation (YMMV).. You can refer to this document [3] for step by step instructions.

Another solution would be to have your app hosted on the private repository and then mirror/clone that repository using Google Cloud Source Repositories [4], run npm install and deploy. As explained in this StackOverflow post, [5], you will have to create a cloudbuild.yaml file on the root folder (same folder the app.yaml file is located) :

steps:
# NPM install
- name: 'gcr.io/cloud-builders/npm'
  args: ['install']
#Test
- name: 'gcr.io/cloud-builders/npm'
  args: ['test']
#Deploy
- name: "gcr.io/cloud-builders/gcloud"
  args: ["app", "deploy"]

You will then have to mirror the private Bitbucket repository to Cloud Source Repository [4], create a Cloud Build Trigger to automate deployment when new code has been pushed to the repository, and then finally pushing the folder containing your application to the repository.

Upvotes: 3

Related Questions