CODEWITHSUNDEEP
azureazure-devopsazure-data-factory
Jefferr
Jefferr

Reputation: 11

Selective releases in Azure Data Factory and DevOps

We have 2 Data factories (dev and prd) with an Azure DevOps Git integration. All setup for CI/CD as mentioned in https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment:

An additional requirement we would like to add, is the possibity to perform selective releases to our prd Data Factory. F.e.: New development A was published to our adf_publish branch and the validation of new development A is still in progress. Meanwhile, new request B needs to be released to ADF-prd as soon as possible (not as a hotfix). However, this is currently blocked by development A, which still has to be validated. So we would like to have the possibility to only release request B to our adf-prd and exclude development A from that release (as it's not validated yet).

Unfortunately, this feature is not supported in Data Factory itself (https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment). We are looking for a solution in Azure DevOps, but so far without luck. Does anyone has a solution for this?

Thanks in advance, Kind regards, Jef

Upvotes: 1

Views: 1027

Answers (2)

Greg
Greg

Reputation: 1

Unfortunately, this is the CI/CD behavior for ADF. Since there is no selective publish, until this moment, there is no way to achieve this using a different branching model and also supported by Microsoft. The way I found to deal with those parallel developments is make use of feature flags (global parameters + if conditions inside pipelines). This is the Microsoft recommended solution, which I don't see as a solution, but a "workaround" for something that they haven't figure out yet.

The other option is having an ADF for each development squad and treat every sprint as a release. This will probably avoid concurrency to promote release to QA and Production.

Hope it helps.

Upvotes: 0

Trent Tamura
Trent Tamura

Reputation: 1145

There is now a way to create an ADF build artifact from a branch and using that artifact to deploy to upper environments, rather than manually clicking Publish in the ADF UX. Using this feature, you could have a breakdown of branches like so:

feature branches (for devlopment)

develop branch (for things not validated yet)

main branch (for things validated or that need to be published to your physical ADF)

using this architecture you could develop a pipeline in feature branch, push it to develop branch where it could have an automated CI build pipeline that generates artifacts, this could either be deployed to a DEV ADF or just used as a validation step, once things are validated in your develop branch, you can do a PR to your main branch which deploys to your DEV or TEST ADF.

In this example, your New Development A would be sitting in the develop branch while your new request B would be pushed directly to your main branch and that would trigger an automated build and deployment to whatever ADF you want...

Please see this StackOverflow Post that I answered on already, it is very thorough.

Azure Data Factory deployments with improved CI/CD

Upvotes: 1

Related Questions