CODEWITHSUNDEEP
azuresnowflake-cloud-data-platformazure-data-factoryazure-data-lakeazure-data-lake-gen2
Bilal Shafqat
Bilal Shafqat

Reputation: 821

Azure Data Factory V2 Copy Activity is not cleaning/deleting staged files/containers after copying data to the destination (Like: Snowflake)

Azure Data Factory V2 Copy Activity is not cleaning/deleting staged files/containers after copying data to the destination (Like: Snowflake).

Is this a bug? In my ADF, the container level is set for the staging area and I end up with multiple GUID containers. Some containers are even having metadata files.

According to Microsoft, all the temporary files/folders/containers created by Azure Data Factory in the Staging Area should be automatically deleted by the ADF after the Copy is successful.

Please guide.

Upvotes: 0

Views: 797

Answers (1)

Pratik Lad
Pratik Lad

Reputation: 8402

According to Microsoft, all the temporary files/folders/containers created by Azure Data Factory in the Staging Area should be automatically deleted by the ADF after the Copy is successful.

For stagging we have 2 options for linked service:

  1. Azure Blob Storage.
  2. Azure Data Lake Service.
  • Azure Blob Storage. -- If you selected Blob storage for staging it will delete or cleanup all folders which are created at the time of staging. -- As blob storage can't keep empty folders in it.

Blob storage After stagging: enter image description here

  • Azure Data Lake Service. -- If you selected ADLS for staging it will just delete or cleanup the data file which is created at the time of staging. -- As you can keep empty folders in an ADLS.

ADLS After stagging: enter image description here

Container is not get deleted in either of the storage accounts.

Upvotes: 2

Related Questions