CODEWITHSUNDEEP
pythonazuresubprocessblobazcopy
Buzzkillionair
Buzzkillionair

Reputation: 379

How to run a blob data transfer through .py

I have tried to create a program to run in my VM to make it so I can do a transfer of data from a directory to my azure blob storage account. Whenever I run the command outside of the program (On the command line) it works, however, if I run the program that contains a subprocess that runs the command, it does not work.

Here is what I send over the command line that works:

sudo ./azcopy cp "/directory/subdirectory" "https://myblob.blob.core.windows.net/container[SAS]" --recursive=true

This completes the data transfer.

When I put it into a program, I ran into many issues.

Current code:

import subprocess
import os
import sys
try:
    key = ('SAS')
    file_path = ('/directory/subdirectory')
    full_link = ('"https://myblob.blob.core.windows.net/' + key + '"')
    transfer = subprocess.check_output(['azcopy', 'cp', file_path,
                                        full_link,
                                        '--recursive=true'], stderr=subprocess.STDOUT)
    print('Transfer Complete.')
# except subprocess.CalledProcessError as e:
#     raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
except EOFError as error:
    #Output for EOF error, would be caused by missing SAS
    print('Error with SAS')
except Exception as e:
    #When an unexpected error has occured.
    print(str(e) + 'Unknown error has occured')
exit(0)

Output:

Command '['azcopy', 'cp', '/directory/subdirectory', '"https://myblob.blob.core.windows.net/[SAS]"', '--recursive=true']' 
returned non-zero exit status 1Unknown error has occured

If I re-add the except statement I have in the code that is currently commented out, I get this error:

Traceback (most recent call last):
  File "data_transfer.py", line 11, in <module>
    '--recursive=true'], stderr=subprocess.STDOUT)
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['azcopy', 'cp', 'datadrive/peeled-images', '"https://myblob.blob.core.windows.net[SAS]"', '--recursive=true']' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "data_transfer.py", line 14, in <module>
    raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
RuntimeError: command '['azcopy', 'cp', '/directory/subdirectory', '"https://myblob.blob.core.windows.net/[SAS]"', '--recursive=true']' return with error (code 1): b'\nfailed to parse user input due to error: the inferred source/destination combination is currently not supported. Please post an issue on Github if support for this scenario is desired\n'

All help is much appreciated

Upvotes: 1

Views: 1688

Answers (1)

Buzzkillionair
Buzzkillionair

Reputation: 379

The answer to this was to change the following line:

subprocess.check_output(['azcopy', 'cp', '/directory/subdirectory',
                 full_link, '--recursive=true'], stderr=subprocess.STDOUT)

The change needed was:

subprocess.call(['azcopy', 'cp', '/directory/subdirectory',
                 full_link, '--recursive=true'], stderr=subprocess.STDOUT)

This is because this was meant to run and execute a program, not necessarily provide a specific output.

Upvotes: 1

Related Questions