CODEWITHSUNDEEP
yamlazure-pipelinesazure-pipelines-yaml
Mike0684
Mike0684

Reputation: 86

Can't checkout the same repo multiple times in a pipeline

I have self-hosted agents on multiple environments that I am trying to run the same build/deploy processes on. I would like to be able to deploy the same code from a single repo to multiple systems concurrently. Thus, I have created an "overhead" pipeline, and several "processes" pipeline templates. Everything seems to be going very well, except for when I try to perform checkouts of the same repo twice in the same pipeline execution. I get the following error:

An error occurred while loading the YAML build pipeline. An item with the same key has already been added.

I would really like to be able to just click ONE button to trigger a main pipeline that calls all the templates requires and gives the parameters needed to get all my jobs done at once. I could of course define this "overhead" pipeline and then queue up as many instances as I need of it per systems that I need to deploy to, but I'm lazy, hence why I'm using pipelines!

As soon as I remove the checkout from Common.yml, the validation succeeds without any issues. If I keep the checkout in there but only call the Common.yml once for the entire Overhead pipeline, then it succeeds without any issues as well. But the problem is: I need to pull the contents of the repo to EACH of my agents that are running on completely separate environments that are in no way ever able to talk to each other (can't pull the information to one agent and have it do some sort of a "copy" to all the other agent locations.....).

Any assistance is very much welcomed, thank you!

The following is my "overhead" pipeline:

# azure-pipelines.yml
trigger:
  none

parameters:
  - name: vLAN
    type: string
    default: 851
    values:
      - 851
      - 1105
  
stages: 
  - stage: vLAN851
    condition: eq('${{ parameters.vLAN }}', '851')
    pool:
      name: xxxxx
      demands:
      - vLAN -equals 851
    jobs:
    - job: Common_851
      steps:
      - template: Procedures/Common.yml
    - job: Export_851
      dependsOn: Common_851
      steps:
      - template: Procedures/Export.yml
        parameters:
          Server: ABTS-01
  - stage: vLAN1105
    condition: eq('${{ parameters.vLAN }}', '1105')
    pool:
      name: xxxxx
      demands:
      - vLAN -equals 1105
    jobs:
    - job: Common_1105
      steps:
      - template: Procedures/Common.yml
    - job: Export_1105
      dependsOn: Common_1105
      steps:
      - template: Procedures/Export.yml
        parameters:
          Server: OTS-01

And here is the "Procedures/Common.yml":

steps:
- checkout: git://xxxxx/yyyyy@$(Build.SourceBranchName)
  clean: true
  enabled: true
  timeoutInMinutes: 1
- task: UsePythonVersion@0
  enabled: true
  timeoutInMinutes: 1
  displayName: Select correct version of Python
  inputs:
    versionSpec: '3.8'
    addToPath: true
    architecture: 'x64'
- task: CmdLine@2
  enabled: true
  timeoutInMinutes: 5
  displayName: Ensure Python Requirements Installed
  inputs:
    script: |
      python -m pip install GitPython

And here is the "Procedures/Export.yml":

parameters:
  - name: Server
    type: string

steps:    
- task: PythonScript@0
  enabled: true
  timeoutInMinutes: 3
  displayName: xxxxx
  inputs:
    arguments: --name "xxxxx" --mode True --Server ${{ parameters.Server }}
    scriptSource: 'filePath'
    scriptPath: 'xxxxx/main.py'

Upvotes: 3

Views: 2047

Answers (2)

wtbman
wtbman

Reputation: 11

I couldn't get it to work with expressions but I was able to get it to work using repository resources following the documentation at: https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops

    resources:
      repositories:
      - repository: MyGitHubRepo # The name used to reference this repository in the checkout step
        type: git
        name: MyAzureProjectName/MyGitRepo  
        ref: $(Build.SourceBranch) 

    trigger:
    - main

    pool:
      vmImage: 'ubuntu-latest'
    #some job
    steps:
    - checkout: MyGitHubRepo
    #some other job     
    steps:
    - checkout: MyGitHubRepo

    - script: dir $(Build.SourcesDirectory)

Upvotes: 1

Simon Hill
Simon Hill

Reputation: 330

I managed to make checkout work with variable branch names by using template expression variables ${{ ... }} instead of macro syntax $(...) variables.

The difference is that, template expressions are processed at compile time while macros are processed at runtime.

So in my case I have something like:

- checkout: git://xxx/yyy@${{ variables.BRANCH_NAME }}

For more information about variables syntax : Understand variable syntax

Upvotes: 1

Related Questions