CODEWITHSUNDEEP
pythonpython-3.xgoogle-cloud-platform
Владимир Кузовкин
Владимир Кузовкин

Reputation: 622

create a new bucket inside bucket in google storage using python

I need to create a folder inside folder in google clous storage using python. I know how to create one folder:

bucket_name = 'data_bucket_10_07'

# create a new bucket
bucket = storage_client.bucket(bucket_name)
bucket.storage_class = 'COLDLINE' # Archive | Nearline | Standard
bucket.location = 'US' # Taiwan
bucket = storage_client.create_bucket(bucket) # returns Bucket object


my_bucket = storage_client.get_bucket(bucket_name)

when I try to change bucket_name = 'data_bucket_10_07' to bucket_name = 'data_bucket_10_07/data_bucket_10_07_1' I got an error:

google.api_core.exceptions.BadRequest: 400 POST https://storage.googleapis.com/storage/v1/b?project=effective-forge-317205&prettyPrint=false: Invalid bucket name: 'data_bucket_10_07/data_bucket_10_07_1'

How should I solve my problem?

Upvotes: 0

Views: 1093

Answers (2)

Walter Zielenski
Walter Zielenski

Reputation: 78

Other comments do a great job of detailing that nested buckets are not possible, but they only suggest at the following shortly: GCS does not rely on folders, and only presents contents with a hierarchical structure on the web UI for ease of use.

From the documentation:

Cloud Storage operates with a flat namespace, which means that folders don't actually exist within Cloud Storage. If you create an object named folder1/file.txt in the bucket your-bucket, the path to the object is your-bucket/folder1/file.txt. There is no folder1 folder, just a single object with folder1 as part of its name.

So if you'd like to create a "folder" for organization and immediately place an object in it, name your object with the "folders" ahead of the name, and GCS will take care of 'creating' them if they don't already exist.

Upvotes: 0

al-dann
al-dann

Reputation: 2725

As John mentioned in the comment, it may not be ontologically possible to have a bucket inside a bucket.

See Bucket naming guidelines for documentation details.

In nutshell:

There is only one level of buckets in a global namespace (thus the bucket name is to be global unique). Everything beyond the bucket name - belongs to an object name.

For example, you can create a bucket (let's guess the name is not already in use) like data_bucket_10_07. In that case, it may look like gs://data_bucket_10_07

Then, you probably would like to store some objects (files) in a such way, that it looks like a directory hierarchy, so, let say there are /01/data.csv object and /02/data.csv object. Where the 01 and 02 should presumably semantically reflect some date.

Those /01/ and /02/ elements - are essentially beginning parts of the object names (or prefixes for the objects in other words).

So far the bucket name is gs://data_bucket_10_07

The object names are /01/data.csv and /02/data.csv

I would suggest checking Object naming guidelines documentation where those ideas are described much better then I can do in one sentence.

Upvotes: 1

Related Questions