CODEWITHSUNDEEP
python-3.xazureazure-blob-storage
derek
derek

Reputation: 10217

azure container_client delete_blobs causes oom

I am using Azure blob SDK python version. Here is how I do blobs deletion:

blobs = container_client.list_blobs(name_starts_with="myprefix") 
container_client.delete_blobs(*blobs)

if blobs here return a large amount of blob objects, the above code crashes. What is the standard practice here? Are there other ways to do batch deletion?

Update:

reply to @Ivan Yang: This is slightly different from your solution. I ran it but got error

There is a partial failure in the batch operation.

blobs = container_client.list_blobs(name_starts_with="myprefix")
blobs_list = list(blobs)
i in range(0, len(blobs_list), 10):
    container_client.delete_blobs(*blobs_list[i: i+10])

Upvotes: 0

Views: 884

Answers (1)

Ivan Glasenberg
Ivan Glasenberg

Reputation: 29950

You'd better specify how many blobs you are trying to delete by using delete_blobs method. Then it's easier for debug.

As a workaround, you can fetch a certain number(like 10 blobs) of blobs each time, then delete until the continuation token is null.

Here is the sample code:

#define a continuation token
continuation_token = None

while True:
    #fetch 10 blobs each time
    blob_list = container_client.list_blobs(name_starts_with="xxx",results_per_page=10).by_page(continuation_token=continuation_token)
    list_segment=[blob.name for blob in list(next(blob_list))]
    container_client.delete_blobs(*list_segment)
    
    continuation_token = blob_list.continuation_token

    if not continuation_token:
        break

Upvotes: 2

Related Questions