Manage dependencies in Azure YAML Devops Pipelines

Question

Our solution consists of few microservices calling and supporting each other.

To simplify, please consider this dependency graph:

 MS1 --> MS2
     --> MS3 --> MS4

• MS1 depends on MS2 and MS3
• MS3 depends on MS4
• MS4 is independent

GOAL: Zero downtime during deployment

Currently we are analyzing the possible ways to solve the couple scenarios described bellow:

1. Deploy all microservices in order, to ensure that all End-To-End tests passes. This means to deploy first MS4, then MS3, MS2, MS1, then run the tests (all of this in slots), then switch the slots if everything passes

2. Deploy any service individually (the others have not changed at all), run the tests (slots again), then switch the slot if everything succeeds

Our first approach is to have a single (big) pipeline which has separate stages per microservice and checks if that microservice have changed to deploy it. If no change is detected for the microservice, then we would like to cancel the Stage and proceed with the next one. This pipeline contain templates for each stage, as for example:

- stage: MS1
  jobs:
  - job: CheckMS1Changes
    steps:
    - template: templates/ms1-check-changes.yml

  - job: BuildMS1
    dependsOn: CheckMS1Changes
    displayName: Build MS1
    - template: templates/ms1-build.yml

  - job: ReleaseMS1
    dependsOn: BuildMS1
    displayName: Release MS1
    - template: templates/ms1-release.yml

We think that this will cover the described scenarios. The "cancel command" should be placed inside the templates/ms1-check-changes.yml file

The problem is that we have not found in documentation how to cancel a complete Stage. Which makes me think that maybe our complete approach is wrong. We also have not found how to cancel a Job or group of jobs, because we also have doubts if we should have stages per microservice or not.

You can see, we are new to this stuff.

Could you give some advice on what could be a good strategy for the described scenarios?

Answer 1

Based on your pipeline. I think you can move the CheckChanges jobs in a separate stage. And then use the logging commands ##vso[task.setvariable variable=MS1;isOutput=true]true to set an output flag variable (ie. MS1) which indicts if changes are detected for each microservice. Then you can use these flags in the condition expressions dependencies.dependencyStageName.outputs['dependencyStageJobName.taskName.varialbeName']

Than you can make following stages dependsOn this stage. And add conditions to decide to skip or run this stage. See below simple example:

stages:
- stage: ChangeStage
  pool: 
    vmImage:  windows-latest
  jobs:
  - job: ChangeJob
    steps:
    - powershell: |
        echo "##vso[task.setvariable variable=MS1;isOutput=true]true" #set MS1 flag to true if changes made to MS1 
        echo "##vso[task.setvariable variable=MS2;isOutput=true]true"
        echo "##vso[task.setvariable variable=MS3;isOutput=true]true"
      name: ChangeTask

- stage: MS3Stage
  dependsOn: ChangeStage
  condition: eq(dependencies.ChangeStage.outputs['ChangeJob.ChangeTask.MS3'], 'true')
  pool: 
    vmImage:  windows-latest
  jobs:
  - template: ...

- stage: MS2Stage
  dependsOn: 
  - MS3Stage
  - ChangeStage
  condition: |
    and 
    (
     eq(dependencies.ChangeStage.outputs['ChangeJob.ChangeTask.MS2'], 'true'),
     in(dependencies.MS3Stage.result, 'Succeeded', 'Canceled', 'Skipped')
    )
  pool: 
    vmImage:  windows-latest
  jobs:
  - template: ...   

- stage: MS1Stage
  dependsOn: 
  - MS2Stage
  - ChangeStage
  condition: |
     and 
    (
     eq(dependencies.ChangeStage.outputs['ChangeJob.ChangeTask.MS1'], 'true'),
     in(dependencies.MS2Stage.result, 'Succeeded', 'Canceled', 'Skipped')
    )
  pool: 
    vmImage:  windows-latest
  jobs:
  - template: ...  
  

In above pipeline. The top stage(ie. ChangeStage) will first run and check if the changes are made to thte microservices and set the output variable to true accordingly.

MS2Stage depends on MS3Stage. And the condition for MS2Stage as below: Which means MS2Stage will only run on the condition of the output flag MS2 is true and MS3Stage is succeeded, skipped or canceled.

MS3Stage and MS1Stage are similar with MS2Stage.

condition: |
    and 
    (
     eq(dependencies.ChangeStage.outputs['ChangeJob.ChangeTask.MS2'], 'true'),
     in(dependencies.MS3Stage.result, 'Succeeded', 'Canceled', 'Skipped')
    )