CODEWITHSUNDEEP
azure-devopsazure-pipelines
chris polzer
chris polzer

Reputation: 3367

Azure DevOps Container Jobs: Cache Container for multiple Jobs?

Dear assorted Developers,

in azure pipeline's container jobs, for every job containers get pulled from registry, even if the same container is used for multiple jobs.

Of course in case the images are really small, this is no problem, but in case anyone is intending to build with the same image which is covering the vscode local development - this can use up more time than the actual build.

So has anyone solved caching the container?

Here is an example:

# in this example, all jobs use the same container.

# in stage 1, the jobs are started serial, so job 2 only starts if 
# job 1 is done -> and the image is downloaded for both jobs independently

# in stage 2, the jobs are started in parallel, 
# and the image is downloaded for both jobs in the stage independently

trigger:
  batch: true
  branches:
    include:
      - "*"


resources:
  containers:
    - container: ubuntu
      image: ubuntu:18.04

stages:
  - stage: STAGE1
    jobs:
     - job: PrintInfoStage1Job1
       container: ubuntu
       steps:
          - script: |
              echo "THIS IS STAGE 1, JOB 1"
            displayName: "JOB 1"
     - job: PrintInfoStage1Job2
       dependsOn: PrintInfoStage1Job1
       container: ubuntu
       steps:
          - script: |
              echo "THIS IS STAGE 1, JOB 2"
            displayName: "JOB 2"

  - stage: STAGE2
    dependsOn: STAGE1
    jobs:
      - job: PrintInfoStage2Job1
        dependsOn: []
        container: ubuntu
        steps:
          - script: |
               echo "THIS IS THE STAGE 2, JOB 1"
            displayName: "JOB 1"

      - job: PrintInfoStage2Job2
        container: ubuntu
        dependsOn: []
        steps:
          - script: |
               echo "THIS IS THE STAGE 2, JOB 2"
            displayName: "JOB 2"

Upvotes: 3

Views: 2263

Answers (1)

Mengdi Liang
Mengdi Liang

Reputation: 19026

Azure DevOps Container Jobs: Cache Container for multiple Jobs?

Initially, our design and develop idea is mostly considering for the security and consistency reasons, it should be a fresh image each time. Now, we have received many feature request about hoping support cache image which same with yours from lots of developers. Now, considering the disadvantage of this design idea, it would let developers wasting too much time to wait for the image pulled down. If the image can be cached, it can greatly improve the efficiency of the build.

Now, the bulk of the actual caching work about this feature has been developed done by our Azure Artifacts Team. Since the latest process I got from that team is before we can release this feature in azure devops, there are some work we need to do, which about around security to make sure that the cache can't be used as an attack vector. Once this is done we will launch a customer preview. It would be deployed recently.

Please see our Roadmap: Speed up pipeline with caching to track its develop and release process. You can also track this blog which published by the azure artifacts PM. Also, you can follow and monitor this PR.

Until now, there's no much better work around to improve this. Even use the Cache task to perform its thing in combination with the Docker save/load respective operations pretty much matched that of downloading the base image/layers from a public registry.

I will still monitor this feature develop process. Once the PR finished and the feature code deployed to all regions, even it released as a preview feature, I will update this answer to let you and other SO users know.

Upvotes: 4

Related Questions