CODEWITHSUNDEEP
google-app-enginegoogle-cloud-storage
Hugues
Hugues

Reputation: 653

Google cloud storage - determine directory size for each user

I have created one bucket which will be used to store data for all my users. The structure of the directories is as follows :

--> BUCKET_NAME/USER_IDENTIFIER/CREATION_DATE/OBJET_ID

==> Is it possible with gsutil or through the API to get the overall size for each of the users ?

For example, if USER X is identified by "USER_1", I'd like to get the size of all objects under BUCKET_NAME/USER_1

Would be great if someone has an idea how to implement this ? Using standard file name, it would be rather easy but I couldn't find an easy way to implement this with Google Cloud Storage.

Thanks,

Hugues

Upvotes: 0

Views: 3255

Answers (4)

Abhishek
Abhishek

Reputation: 69

This script will give you the size of all individual sub-folder for a given parent directory, will suffice for the above usecase.

gsutil ls -l gs://bucket_name/folder_name | xargs -I{} gsutil du -sh  {}

You can pipeline the output of the script to a text file using below script:

gsutil ls -l gs://bucket_name/folder_name | xargs -I{} gsutil du -sh  {} > $HOME/file.txt 2>&1

Upvotes: 0

roberto.cr
roberto.cr

Reputation: 322

You can now use a native gsutil command for that:

To print the total number of bytes in a bucket, in human-readable form:

gsutil du -ch gs://bucketname

source: https://developers.google.com/storage/docs/gsutil/commands/du

It returns a total per "folder" and a global total, which is super convenient!

Upvotes: 1

Mike Schwartz
Mike Schwartz

Reputation: 12155

If user names are regularly structured you could write a script like this:

for user_id in {0..100}; do;
    echo USER ${user_id}: $(gsutil ls -l gs://BUCKET_NAME/$user_id/**) 
done

This will print the number of objects and total data size for each user (assuming your user IDs are simple numeric values).

If you have a large number of objects for each user this process can be very slow, as Shay noted.

Upvotes: 1

Shay Erlichmen
Shay Erlichmen

Reputation: 31928

There isn't an API for that, you will need to keep the bookkeeping per user your self.
You can request all the files under a certain key and sum the total files size but this process is (very) slow.

Upvotes: 1

Related Questions