CODEWITHSUNDEEP
google-cloud-storage
Echo
Echo

Reputation: 405

gsutil copy, how to specify max number of objects to copy?

I need to move files from one bucket to another in GCS for processing, each time I just need to copy and process certain amount of files, say 100, how can I configure it in the command line?

gsutil -m mv gs://bucket1/file001.json gs://bucket2/

Thanks for your help.

Upvotes: 0

Views: 730

Answers (1)

Mike Schwartz
Mike Schwartz

Reputation: 12145

You could write a script that builds a list of all objects to be copied and then walks through them 100 at a time, calling gsutil mv one at a time. That would lose parallelism compared to running gsutil -m mv, so you could run each of those gsutil mv commands in the background and then wait for all 100 to complete before moving on to the next batch.

Upvotes: 1

Related Questions