CODEWITHSUNDEEP
google-cloud-storagegsutil
Zaki Indra Sukma
Zaki Indra Sukma

Reputation: 57

How to correctly use `gsutil -q stat` in scripts?

I am creating a KSH script to check whether a subdirectory is exist on GCS bucket. I am writing the script like this:

#!/bin/ksh

set -e
set -o pipefail

gsutil -q stat ${DESTINATION_PATH}/
PATH_EXIST=$?
if [ ${PATH_EXIST} -eq 0 ] ; then
   # do something
fi

Weird thing happens when the ${DESTINATION_PATH}/ does not exist, the script exit without evaluating PATH_EXIST=$?. If ${DESTINATION_PATH}/ is exist, the script will run normally as expected.

Why does that thing happen? How can I do better?

Upvotes: 0

Views: 2917

Answers (2)

Damian Kober
Damian Kober

Reputation: 1

you can do a short version of this like this

if gsutil -q stat "gs://some-bucket/some-object"
then  
    echo "exists"
else
    echo "does not exist"
fi

Upvotes: 0

llompalles
llompalles

Reputation: 3176

The statement set -e implies that your script will be exited if a command exits with a non-zero status.

The gsutil stat command can be used to check wheter an object exists:

gsutil -q stat gs://some-bucket/some-object

It has an exit status of 0 for an existing object and 1 for a non-existent object.

However it is advised against to use it with subdirectories:

Note: Unlike the gsutil ls command, the stat command does not support operations on sub-directories. For example, if you run the command:

gsutil -q stat gs://some-bucket/some-subdir/

gsutil will look for information about an object called some-subdir/ (with a trailing slash) inside the bucket some-bucket, as opposed to operating on objects nested under gs://some-bucket/some-subdir/. Unless you actually have an object with that name, the operation will fail.

The reason because your command is not failing when your ${DESTINATION_PATH}/ exists is because if you create the folder using the Cloud Console i.e the UI, then a placeholder object will be created with its name. But let me be clear, folders don't exist in Google Cloud Storage, they are just a visualization of the bucket objects hierarchy.

So if you upload an object named newFolder/object to your bucket and the newFolder does not exists, it will be "created" but your gsutil -q stat ${DESTINATION_PATH}/ will return exit code 1. However if you create the folder using the UI and run the same command it will return exit 0. Thus follow the documentation, and avoid using it for checking if a directory exists.

Instead if you want to check whether a subdirectory exists just check if it contains any object inside:

gsutil -q stat ${DESTINATION_PATH}/*

Which will return 0 if any object is in the subdirectory and 1 otherwise.

Upvotes: 3

Related Questions