CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-storage
tikael
tikael

Reputation: 449

check if google storage bucket is successfully created/is not empty using bash with large bucket size

Currently I am using:

is_created=$(gsutil du -s ${bucket_name} 2> /dev/null || true)
if [ -z "${is_created}" ]; then
    gsutil mb -p ${GCLOUD_PROJECT} -c regional -l ${gcloud_region} ${bucket_name}
fi

Yet since my bucket size is large and it takes a long time to load the result. Is there another way around?

PS I've tried:

gsutil -m du -s

and did not see a noticeable difference.

Upvotes: 0

Views: 1312

Answers (1)

Dan Cornilescu
Dan Cornilescu

Reputation: 39824

The gsutil du command is intended to obtain info about the size of the bucket objects. Sure, as a side effect you can use it to determine if the bucket is accessible and not empty (or not).

Note that I'm saying accessible, not created, as you won't be able to tell the difference between bucket not created, created but not accessible (due to permissions or some failure of some sort) or accessible but empty. Except for the last/empty case attempting to create the bucket will fail.

From performance/duration perspective gsutil du isn't that great, it can be quite slow in buckets with a lot of objects in them as it spends time with size calculations, which are irrelevant for your goal.

One thing to try is the gsutil ls command instead, intended to obtain just the list of objects in the bucket, which typically uses less CPU than gsutil du (no size info collection/calculations). Use it without options to prevent collecting additional object info unnecessarily, just the object name should be enough for the empty check.

Something along these lines maybe:

missing=$(gsutil ls ${bucket_name} |& grep BucketNotFound | wc -l)
if [ ${missing} == 1 ]; then
    gsutil mb -p ${GCLOUD_PROJECT} -c regional -l ${gcloud_region} ${bucket_name}
fi

Or, even faster on buckets with many objects:

created=$(gsutil ls -p ${GCLOUD_PROJECT} | grep ${bucket_name} | wc -l)
if [ ${created} == 0 ]; then
    gsutil mb -p ${GCLOUD_PROJECT} -c regional -l ${gcloud_region} ${bucket_name}
fi

Upvotes: 1

Related Questions