CODEWITHSUNDEEP
sql-serverazure-blob-storageazcopy
nick
nick

Reputation: 333

Azcopy move or remove specific files from SQL

I need to move files from one blob to another. Not copy. I need to move; meaning when I move from A to B then files go from A to B and nothing is there in A. Which is so basic but not possible in Azure Blob. Please let me know if its possible. I am using it from SQL Server using AzcopyVersion 10.3.2

Now because of this, I need to copy files from A to B and then remove files form A. There are 2 problems.

1) I only want certain files to go from A to B.

DECLARE @Program varchar(200) = 'C:\azcopy.exe'
DECLARE @Source varchar(max) = '"https://myblob.blob.core.windows.net/test/myfolder/*?SAS"'
DECLARE @Destination varchar(max) = '"https://myblob.blob.core.windows.net/test/archive?SAS"'

DECLARE @Cmd varchar(5000)
SELECT @Cmd = @Program +' cp  '+ @Source +' '+ @Destination + ' --recursive'
PRINT @cmd

EXECUTE master..xp_cmdshell @Cmd

So When I type myfolder/* then it will take all the files. When I try myfolder/*.pdf, it says failed to parse user input due to error: cannot use wildcards in the path section of the URL except in trailing "/*". If you wish to use * in your URL, manually encode it to %2A

When I try myfolder/%2A.pdf OR myfolder/%2Apdf it still gives the error.

INFO: Failed to create one or more destination container(s). Your transfers may still succeed if the container already exists.

But the destination folder is already there. And in the log file it says,

RESPONSE Status: 403 This request is not authorized to perform this operation.

Upvotes: 1

Views: 4455

Answers (1)

Ivan Glasenberg
Ivan Glasenberg

Reputation: 30005

For azcopy version 10.3.2:

1.Copy specific files, like only copy .pdf files: you should add --include-pattern "*.pdf" to your command. And also remember for the @Source variable, remove the wildcard *, so your @Source should be '"https://myblob.blob.core.windows.net/test/myfolder?SAS"'.

The completed command looks like this(please change it to meet your sql cmd):

azcopy cp "https://xx.blob.core.windows.net/test1/folder1?sas" "https://xx.blob.core.windows.net/test1/archive1?sas" --include-pattern "*.pdf" --recursive=true

2.For delete specific blobs, like only delete .pdf files, you should also add --include-pattern "*.pdf" to your azcopy rm command.

And also, there is no move command in azcopy, you should copy it first => then delete it. You can achieve this with the above 2 commands.

Upvotes: 3

Related Questions