CODEWITHSUNDEEP
permissionsgoogle-cloud-storagegsutilgoogle-cloud-sdk
Canovice
Canovice

Reputation: 10173

Removing files from GCS: "gsutil -m rm" throws CommandException: files/objects could not be removed

gsutil -m rm gs://{our_bucket}/{dir}/{subdir}/*

...
Removing gs://our_bucket/dir/subdir/staging-000000000102.json...
Removing gs://our_bucket/dir/subdir/staging-000000000101.json...
CommandException: 103 files/objects could not be removed.

The command is able to find the directory with the 103 .JSON files, and "tries" removing them per the Removing gs://... being output. For what reason might we be receiving CommandException: 103 files/objects could not be removed.?

Perhaps this is a permissions issue with the compute engine not having permission to remove files in our GCS?

Edit: We have a service account JSON in the /config folder of our Airflow project, and that service account is shared to an IAM user with Storage Admin permission. Perhaps having the JSON in the /config folder is not sufficient for assigning permissions to the entire GCP compute engine? I am particularly confused because this server is able to query from our BQ database, and WRITE to GCS, but cannot delete from GCS...

Upvotes: 1

Views: 3100

Answers (1)

Canovice
Canovice

Reputation: 10173

The solution in this link - https://gist.github.com/ryderdamen/926518ddddd46dd4c8c2e4ef5167243d was exactly what we needed:

  • Stop the instance
  • Edit the settings
  • Remove gsutil cache

Upvotes: 2

Related Questions