CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-storagegoogle-cloud-build
sudcha
sudcha

Reputation: 793

Access a file in one bucket by reading the path from another file using gsutil

What is the best way to access a file in one bucket whose path is stored in another file in different bucket?

Most multi fold steps suggest using Cloud Storage for persisting values between steps and writing values of variables into files, but after many attempts, we are unable to achieve this even for a simple use case in Cloud Build setup.

Few of the many things we have tried:

Concatenating steps in Cloud Shell which works in Cloud Shell, but fails in Cloud Build steps:

gsutil cp gs://bucket_1/filepath.txt filepath.txt && filepath=$(<filepath.txt) && gsutil cp gs://bucket_2/filepath local_dest.txt

Cloud Build fails as it doesn't recognize the command "filepath=$(<filepath.txt)".

Cloudbuild.yaml steps (simplified to test). Step1 succeeds, but Step2 fails

- name: gcr.io/cloud-builders/gsutil
  id: 'step1'
  entrypoint: 'gsutil'
  args: ['cp', 'gs://bucket_1/filepath.txt', 'filepath.txt']
  volumes:
  - name: data
    path: /persistent_volume

- name: gcr.io/cloud-builders/gsutil
  id: 'step2'
  entrypoint: 'gsutil'
  args: ['filepath=$(<filepath.txt)', 'gsutil cp gs://anc-android-builds/$BRANCH_NAME/filepath local_dest.txt']
  volumes:
  - name: data
    path: /persistent_volume

Error: CommandException: Invalid command "filepath=$(<filepath.txt)".

We've tried different ways of pushing this, and breaking it down into multiple steps as well, but nothing seems to work.

This must be a simple answer, but we can't seem to figure this out. Please help and advise.

Upvotes: 0

Views: 255

Answers (1)

bhito
bhito

Reputation: 2683

In order to achieve what you're looking for, you need to modify your 2nd step, as for now the entrypoint is expecting a gsutil command but not receiving it straight away. Therefore you need to change to something like:

- name: gcr.io/cloud-builders/gsutil
  id: 'step2'
  entrypoint: 'bash'
  args:
      - '-c'
      - |
      - filepath=$(<filepath.txt) && gsutil cp gs://anc-android-builds/$BRANCH_NAME/filepath local_dest.txt
  volumes:
  - name: data
    path: /persistent_volume

You may need to tweak it a bit more, depending on you exact scenario but this should be the correct path to achieve what you are intending to.

Upvotes: 1

Related Questions