run azuredevops task in all agents of the specified pool

Question

Is there any way to make the pipelie task/job to run in all the agents of the agent pool ?

I am planning to create a release pipeline and scheduled once in every week, which is for cleaning up the build agents from unused docker images which are older than 3 months. so I got the command for the same and created one release pipeline with the "commandline" task with the command "docker image prune --all --filter "until=720h". But here we have multiple agents in the each pool and i have to ensure that this task is executed across all the agents in the specified pools. How can i Achieve this

Answer 1

I had a use case for running cleanup jobs in multiple agent-pools and individual agents in loops. This may be helpful for some users.

Here is the complete code.

parameters:
- name: cleanups
  type: object
  default:
  - projects: iOS
    agents: [
      ios-Agent-1,
      ios-Agent-2,
    ]
  - projects: Android
    agents: [
      android-Agent-1,
      android-Agent-2,
      android-Agent-3,
    ]
  - projects: Cloud
    agents: [
      cloud-Agent-1,
      cloud-Agent-2,
      cloud-Agent-3,
      cloud-Agent-4,
      cloud-Agent-5,
    ]

variables:
  system.debug: false

trigger: none
schedules:
- cron: '1 22 * * 5,3' # At 22:01 on Friday and Wednesday.
  displayName: ADO Agent Cleanup Schedule
  branches:
    include:
    - master
  always: true

jobs:
- ${{ each project in parameters.cleanups }} :
  - ${{ each agent in project.agents}} :
    - job: ${{replace(agent, '-', '_')}}_${{project.projects}}

      variables:
        ${{ if eq( project.projects, 'Cloud') }}:
          agentPool: XX-XX-Ubuntu
        ${{ elseif eq( project.projects, 'iOS') }}:
          agentPool: XX-XX-Mac
        ${{ else }}:
          agentPool: XX-XX-Windows

      pool:
        name: $(agentPool)
        demands: 
        - Agent.Name -equals ${{ agent }}

      steps:
      - checkout: none
      - task: Shellpp@0
        displayName: '${{ project.projects}} Agent Cleanup ${{ agent }}'
        inputs:
          type: 'InlineScript'
          script: |
            echo            
            echo ${{ project.projects}} ${{ agent }} $(agentPool)
            echo
            if [ "$(agentPool)" == "XX-XX-Ubuntu" ]; then
              echo
              echo "Run your Script here for Ubuntu agents"
              echo
            elif [ "$(agentPool)" == "XX-XX-Windows" ]; then
              echo
              echo "Run your Script here for Windows agents"
              echo
            elif [ "$(agentPool)" == "XX-XX-Mac" ]; then
              echo
              echo "Run your Script here for MAC agents"
              echo
            else
              echo
              echo "Nothing to Clean"
              echo
            fi

Notes:

1. Yes, this cleanup is on top of agent maintenance jobs by Azure DevOps.
2. You could make it fully dynamic by getting agents lists from Azure DevOps REST API in place of parameters use in the code above.

Answer 2

While @GeralexGR's answer works and I've gone that route in the past, it can get really annoying once you try to scale up your environment.

That's why I switched to deployment groups for "agent management". Obviously you will need to install another agent on every machine. But that can also be an advantage: the ratio of Servers to agents doesn't have to be 1:1, we run multiple agents per machine therefore if I were to list every agent in this case docker would prune everything twice.

With deployment groups you only need to add the agent (1 per machine) to the group and all pipelines "maintaining" that group of agents will automatically target that machine as well for future runs. The main disadvantage in my opinion is, that you cannot use deployment groups in yaml (yet?!). So you need to use the GUI to configure deployment group tasks in the release pipelines.

Answer 3

You could create a pipeline with multiple Stages and run every stage on each agent.

trigger:
- main

pool:
  vmImage: ubuntu-latest

stages:
- stage: agent1
  pool: 
   name: AgentPoolName
   demands:
    - agent.name -equals agent1
  jobs:
  - job: docker1
    steps:
    - script: docker image prune --all --filter "until=720h

- stage: agent2
  pool: 
   name: AgentPoolName
   demands:
    - agent.name -equals agent2
  jobs:
  - job: docker2
    steps:
    - script: docker image prune --all --filter "until=720h