CODEWITHSUNDEEP
google-cloud-platformbucketgsutil
markzhu
markzhu

Reputation: 123

How to know path "gs://bucket1/folder_x" existing or not in GCP bucket

Is there a '''gsutil''' command can tell me if the path '''gs://bucket1/folder1_x/folder2_y/''' existing or not? Is there a '''ping''' command in gsutil? I use Jenkins parameters folder_x and folder_y which value input by user, and joined by pipeline. Currently, if the dir does exist, the pipeline will show success. But if the path is wrong, the pipeline will be interrupted and shows failure.

Tried use gsutil stat and gsutil -q stat, it can test '''gs://bucket1/folder1_x/folder2_y/file1''', but not for dir.

'''groovy

pipeline {

stages {

    stage('Check existing dirs') {

        steps {
            script{

                if (params['Action'] =="List_etl-output") {

                    def Output_Data="${params['Datasource']}".toString().split(",").collect{"\"" + it + "\""}
                    def Output_Stage="${params['Etl_Output_Stage']}".toString().split(",").collect{"\"" + it + "\""}
                    for (folder1 in Output_Data) {
                        for (folder2 in Output_Stage) {
                        sh(script:  """

                           gsutil ls -r gs://bucket1/*/$Data/$Stage
                          
                      """ )

                    }
                }
            }
        }
      }
    }

} }

'''

I was use gsutil to check if the path gs://bucket1/*/$Data/$Stage available or not. The $Data and $Stage are given by user input, the Jenkins pipeline interrupted when the path not available. I want gsutil can skip the wrong path when it's not available.

Upvotes: 0

Views: 1884

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 76018

The directory doesn't exist in Cloud Storage. It's a graphical representation. All the blob are stored at to bucket root and their name is composed of the full path (with / that you interpret as directory, but not). It's also for this that you can only search on prefix.

To answer your question, you can use this latest feature: search on the prefix. If there is 1 element, the folder exist BECAUSE there is at least 1 blob with this prefix. Here an example in Python (I don't know your language, I can adapt it in several language if you need)

    from google.cloud import storage
    client = storage.Client()
    bucket = client.get_bucket('bucket1')
    if len(list(bucket.list_blobs(prefix='folder_x/'))):
      print('there is a file in the "directory"')
    else:
      print('No file with this path, so no "directory"')

Here the example in Groovy

import com.google.cloud.storage.Bucket
import com.google.cloud.storage.Storage
import com.google.cloud.storage.StorageOptions

Storage storage = StorageOptions.getDefaultInstance().service
Bucket bucket = storage.get("bucket1")

System.out.println(bucket.list(Storage.BlobListOption.prefix("folder_x/")).iterateAll().size())

Upvotes: 2

Related Questions