Azure-CLI Blob Storage remove multiple blobs

I need to remove multiple files/blobs with the Azure-CLI. But it's not officially supported. So I wrote a for-loop to remove multiple files with the az command. It's look like this:

for i in `az storage blob list --container-name $BUCKET --output table | awk '{print $1}'| sed '1,2d' | sed '/^$/d'` ; do
    if az storage blob delete --name $i --container-name $BUCKET --output table ; then
      echo "Files inside the bucket (container) ${BUCKET} have been erased"
    else
      echo "Unable to erase the files inside the bucket (container) ${BUCKET}." && exit 1
    fi
done

But this doesn't work correctly and I think the loop hangs. But I don't know why.

I hope anyone can help me. Thanks in advance.

Edit:

Thanks for your help. But nothing happen. I'm working on Xubuntu 17.04 and had fresh installed the azure cli.

I get this Message, if I CTRL+C:

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/applicationinsights/channel/SenderBase.py", line 118, in send
  File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 466, in open

    response = self._open(req, data)
  File "/usr/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 1297, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/lib/python3.5/http/client.py", line 1253, in connect
    super().connect()
  File "/usr/lib/python3.5/http/client.py", line 849, in connect
(self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib/python3.5/socket.py", line 703, in create_connection
    sock.connect(sa)
KeyboardInterrupt

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/azure/cli/core/telemetry.py", line 271, in <module>
  File "/usr/lib/python3/dist-packages/azure/cli/core/telemetry.py", line 263, in upload_telemetry
  File "/usr/lib/python3/dist-packages/applicationinsights/TelemetryClient.py", line 55, in flush
  File "/usr/lib/python3/dist-packages/applicationinsights/channel/TelemetryChannel.py", line 71, in flush
  File "/usr/lib/python3/dist-packages/applicationinsights/channel/SynchronousQueue.py", line 39, in flush
  File "/usr/lib/python3/dist-packages/applicationinsights/channel/SenderBase.py", line 118, in send
  File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/usr/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 1297, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/lib/python3.5/http/client.py", line 1253, in connect
    super().connect()
  File "/usr/lib/python3.5/http/client.py", line 849, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib/python3.5/socket.py", line 703, in create_connection
    sock.connect(sa)
KeyboardInterrupt

It's totaly unnecessary where I run this script. I get always the same error :-(

--aruka

Upvotes: 0

Answers (3)

Johan Stenberg - MSFT

Reputation: 124

A couple of tips (unrelated to the root of the issue which is why the delete commands hang) - you don't have to use the sed/awk dance to get the blob name. There is a specific output format (tsv) that is intended to give you a more scripting friendly output (no headers or other distracting things to exclude). Combined with --query, you can then select which fields of the output you are interested in:

az storage blob list -c templates --account-name johanstetemplates --query "[].name" --output tsv

will give you the list of blobs in the given container.

Personally, I'm partial to using xargs if I want to process many items in parallel and control things like how many concurrent requests I make etc.

    az storage blob list -c templates --account-name johanstetemplates --query "[].name" --output tsv | xargs -L1 -P10 -I{} az storage blob show -c templates --account-name johanstetemplates -n {}

(I changed the delete command to show to prevent users from blindly copy/paste the command and get very upset when all their blobs got deleted)

Upvotes: 1

Jason Ye

Reputation: 13954

In my test, your script works for me, maybe we can try this script:

export AZURE_STORAGE_ACCOUNT=jasontest321
export AZURE_STORAGE_ACCESS_KEY=+v6/ZQy74QhIwdvEMbnMV7HMaDaNUAkCDizBTOVdZkBUlw==
export container_name=vhds
for i in `az storage blob list --container-name $container_name --output table | awk '{print $1}'| sed '1,2d' | sed '/^$/d'` ; do
    if az storage blob delete --name $i --container-name $container_name >/dev/null ; then
      echo "Files $i inside the $container_name have been erased"
    else
      echo "Unable to erase the files $i inside the  $container_name." && exit 1
    fi
done

Here is the screenshot about the result:

Update:

#az --version
.
role (2.0.5)
sf (1.0.2)
sql (2.0.4)
storage (2.0.7)
vm (2.0.7)

Python (Linux) 2.7.5 (default, Nov  6 2016, 00:28:07) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

Python location '/root/lib/azure-cli/bin/python'
[root@jasoncli@jasonye ~]# python
Python 2.7.5 (default, Nov  6 2016, 00:28:07) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

In my lab, my root account version is 2.7.5. we should check your python version for your user aruka.

Update2:
We can use Azure portal cloud shell to test this script:

Update3:

root@jasonvm:~# history
    1  echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ wheezy main" |      sudo tee /etc/apt/sources.list.d/azure-cli.list
    2  sudo apt-key adv --keyserver packages.microsoft.com --recv-keys 417A0893
    3  sudo apt-get install apt-transport-https
    4  sudo apt-get update && sudo apt-get install azure-cli
    5  export AZURE_STORAGE_ACCOUNT=jasondisk321
    6  export AZURE_STORAGE_ACCESS_KEY=O1/xxxWRzOlWep78iRcxxx/htgxHImKxxytlsv7gnn0hg1aGvw==
    7  export container_name=vhds
    8  for i in `az storage blob list --container-name $container_name --output table | awk '{print $1}'| sed '1,2d' | sed '/^$/d'` ; do     if az storage blob delete --name $i --container-name $container_name >/dev/null ; then       echo "Files $i inside the $container_name have been erased";     else       echo "Unable to erase the files $i inside the  $container_name." && exit 1;     fi; done
    9  az login -u [email protected] -p xxxx
   10  export AZURE_STORAGE_ACCOUNT=jasondisk321
   11  export AZURE_STORAGE_ACCESS_KEY=O1/2D1UxeGcOlWep78iRcR5PxxgxxxzRbWoytlsv7gnn0hg1aGvw==
   12  export container_name=vhds
   13  for i in `az storage blob list --container-name $container_name --output table | awk '{print $1}'| sed '1,2d' | sed '/^$/d'` ; do     if az storage blob delete --name $i --container-name $container_name >/dev/null ; then       echo "Files $i inside the $container_name have been erased";     else       echo "Unable to erase the files $i inside the  $container_name." && exit 1;     fi; done
   14  export AZURE_STORAGE_ACCOUNT=jasondisk321
   15  export AZURE_STORAGE_ACCESS_KEY=O1/2D1UC0xxx/ztABmMyQFCeGcWRzOlWep78iRcR5Pt/K9yKxMT/htgxHImKqgznzRbWoytlsv7gnn0hg1aGvw==
   16  export container_name=vhds
   17  for i in `az storage blob list --container-name $container_name --output table | awk '{print $1}'| sed '1,2d' | sed '/^$/d'` ; do     if az storage blob delete --name $i --container-name $container_name >/dev/null ; then       echo "Files $i inside the $container_name have been erased";     else       echo "Unable to erase the files $i inside the  $container_name." && exit 1;     fi; done
   18  history

Upvotes: 1

aruka

Reputation: 3

The first command (on my own azure):

$ az storage blob list --container-name testbucket --output table
Name         Blob Type      Length  Content Type    Last Modified
-----------  -----------  --------  --------------  -------------------------
results.csv  BlockBlob         254  text/csv        2017-08-02T22:30:05+00:00

And the version:

$ az --version
azure-cli (2.0.12)

acr (2.0.9)
acs (2.0.11)
appservice (0.1.11)
batch (3.0.3)
billing (0.1.3)
cdn (0.0.6)
cloud (2.0.7)
cognitiveservices (0.1.6)
command-modules-nspkg (2.0.1)
component (2.0.6)
configure (2.0.10)
consumption (0.1.3)
container (0.1.7)
core (2.0.12)
cosmosdb (0.1.11)
dla (0.0.10)
dls (0.0.11)
feedback (2.0.6)
find (0.2.6)
interactive (0.3.7)
iot (0.1.10)
keyvault (2.0.8)
lab (0.0.9)
monitor (0.0.8)
network (2.0.11)
nspkg (3.0.1)
profile (2.0.9)
rdbms (0.0.5)
redis (0.2.7)
resource (2.0.11)
role (2.0.9)
sf (1.0.5)
sql (2.0.8)
storage (2.0.11)
vm (2.0.11)

Python (Linux) 2.7.13 (default, Jan 19 2017, 14:48:08) 
[GCC 6.3.0 20170118]

Python location '/home/aruka/lib/azure-cli/bin/python'

Upvotes: 0

Azure-CLI Blob Storage remove multiple blobs

Answers (3)

Related Questions