CODEWITHSUNDEEP
azureazure-blob-storageazure-data-factory
Madrina Thapa
Madrina Thapa

Reputation: 61

Is there any workaround for using Append blob in Azure Data Factory copy?

I don't want my copy data activity in ADF pipeline to overwrite a blob file but instead, append the new data to it. But ADF only supports Blockblob so is there a way to append data to an existing file?

Upvotes: 3

Views: 3998

Answers (1)

Juani D'Ambrosio
Juani D'Ambrosio

Reputation: 351

You can use Web Activity instead of Copy Activity. Using Blob Service Rest API you could send a PUT request with data you want to append to a blob. You can optionally consume datasets and linked services by this activity.

JSON:

{  
   "name":"AppendDataActivity",
   "type":"WebActivity",
   "typeProperties":{  
      "method":"Put",
      "url":"https://<myAccount>.blob.core.windows.net/<myContainer>/<myBlob><SASToken>&comp=appendblock",
      "headers":{  
         "Content-Type":"application/json" 
      },
      "datasets":[  
         {  
            "referenceName":"<consumedDatasetName>",
            "type":"DatasetReference",
            "parameters":{  
               ...
            }
         }
      ],
      "linkedServices":[  
         {  
            "referenceName":"<consumedLinkedServiceName>",
            "type":"LinkedServiceReference"
         }
      ]
   }
}

Note that Body can only support JSON object or JSON value, JSON array is being fixed but non-JSON values aren't supported. For this workaround you could use an Azure Function as endpoint, where you could append files or whatever you want.

Upvotes: 2

Related Questions