Can I use one AzureDevOps pipeline to run other pipelines?

Question

I would like to have a master pipeline capable of running the pipelines of our system's individual components. I'd also like to be able to run any of those components' pipelines individually. Additionally, some of the component pipelines are configured using yaml, while others are using the classic approach. (I'm not sure if that figures into any possible solutions to this problem.) Those that are configured using yaml typically contain multiple jobs, and I'd need all of the jobs to run in those cases.

Using approach #2 recommended here, I tried the following:

jobs:
- job: build_and_deploy
  displayName: Build and Deploy
  cancelTimeoutInMinutes: 1
  pool:
   name: some-pool
  steps:
  - checkout: self  
  - template: component_one_pipeline.yml 
  - template: component_two_pipeline.yml

I receive an error for the following "unexpected values": trigger, resources, name, variables, and jobs. I'm guessing these aren't allowed in any yaml file referenced in the template step of another pipeline yaml file. As I mentioned above, though, I need these values in their files because we need to run the pipelines individually.

If possible, could someone point me in the direction of how to get this done?

EDIT: I have also tried the approach given here. I was thinking I'd have a master pipeline that essentially did nothing except serve as a trigger for all of the child pipelines that are supposed to run sequentially. Essentially, the child pipelines should subscribe to the master pipeline and run when it's done. I ended up with the following 2 files:

# master-pipeline.yml
trigger: none

pool:
  name: some agent pool

steps:
- script: echo Running MASTER PIPELINE
  displayName: 'Run master pipeline'

#child-pipeline.yml
trigger: none
#- testing-branch  (tried these combinations trying to pick up master run)
#- main

pool:
  name: some agent pool
  
resources:
  pipelines:
  - pipeline: testing_master_pipeline
    source: TestingMasterPipeline
    trigger: true

steps:
- script: echo Running CHILD PIPELINE 1
  displayName: 'Run Child Pipeline 1'

Unfortunately, it's not working. I don't get any exceptions, but the child pipeline isn't running when I manually run the master pipeline. Any thoughts?

Thanks in advance.

Answer 1

As it turns out, I needed to set the pipelines' default branch to the one I was testing on for things to work correctly. The code in my original post works fine.

If you run into this problem and you're using a testing branch to test your pipelines, check out the information here on how to configure your pipeline to listen for triggers on your branch. Apparently a pipeline only listens for them on its default branch. Note: The example in the link uses the "classic" approach to pipeline configuration as an example, but you can reach the same page from a yaml configuration's edit screen by clicking the 3 dots on the right and selecting "Triggers."

Hope this helps someone.

Answer 2

The way that those approaches you linked work, and Azure DevOps build triggering works in general, is that a build completion can trigger another build, and you have to have the trigger in build to be triggered. So:

1. Yaml templates can't have things like triggers, so they won't really help you here (though you can of course split any of the individual pipelines to templates). Triggers are in the main yaml pipeline fail, which references the template-files. So you can't have a individual component pipelines as templates.

2. Yaml pipelines can be chained with the resources-declaration mentioned in the first link. The way this works is that the resource declaration is in the pipeline to be triggered, and you configure the conditions (like branch filters: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/pipeline-triggers?view=azure-devops#branch-filters) to the pipeline to be triggered. For example, in your component pipeline you declare the master pipeline as resource, and set the conditions when the component pipeline will be triggered, like when the master pipeline is run against /release/* -branch. Or just set the trigger to true in order to trigger the component pipeline from any succesful run of the master pipeline. The component pipeline can still have its own pipeline triggers at the start of the pipeline declaration.

3. The classic build definitions can also be chained via edit build definition -> triggers -> build completion (see, for example, here: https://jpearson.blog/2019/03/27/chaining-builds-in-azure-devops/). This works the same way as with yaml pipelines; you configure the conditions for this the classic pipeline to trigger, so add the master pipeline as trigger to the component pipelines. Again, you can also set pipeline triggers for the component pipeline.

The limitation here is, that a classic pipeline can be triggered by an yaml pipeline, but not vice versa. A similar limitation in the yaml resources-declaration; they can't be triggered by a classic pipeline. If you need such triggering, or otherwise find the "native" triggers not to be enough, you can of course shoot an Azure DevOps API call in either type of pipeline to trigger any pipeline. See: https://blog.geralexgr.com/cloud/trigger-azure-devops-build-pipelines-using-rest-api, or just search for the azure devops rest api and associated blog posts that trigger the api with powershell, the rest api -task or by some other means.