CODEWITHSUNDEEP
azure-pipelinesazure-artifactsbuild-serverretention
Dib
Dib

Reputation: 2093

Policy to delete artifacts from file system of dev ops build server

Background:

We have an on-prem Azure DevOps build server with a number of agents. We have recently run out of disk space on it and discovered that although we have 30 days retention policies set up in Azure DevOps there are build artifacts nearly a year old on the file system. I have read the articles linked below, and used them to check our project and organisation retention policies. The age of the artifacts on the server indicate these are being ignored.

Screen Shots:

Below is a screen shot of the project settings for retention.

enter image description here

Below is our release retention settings:

enter image description here

Questions:

Reference:

UPATE: I have now implemented the "maintenance job" for the build agent pool, following the advice in on of the comments.

Upvotes: 2

Views: 70

Answers (2)

Jonathan Dodds
Jonathan Dodds

Reputation: 5163

I assume you are asking about build artifacts (and not build agent working directories). Build artifacts are produced by a build pipeline and are stored either in the Microsoft SQL Server database that is the backing data store for Azure DevOps or on a file share. (Note that the file share doesn't need to be on the same host as the Azure DevOps server and different build pipelines can use different file shares on different host machines.)

There are "global" retention settings at the Organization (aka collection) level. These can be overridden at the Project level. Older versions of Azure DevOps allowed overriding at the pipeline level but newer versions dropped support for this. I'm guessing your screenshot is either of the pipeline or of the project where the current user doesn't have the proper admin rights.

The "Retain" checkbox will retain a build forever regardless of retention policy until the retain is unchecked/removed.

When a release pipeline consumes a build artifact of a build pipeline, the release pipeline creates a "retain" on the build. A set of retention policy settings that aggressively prune builds will have no effect if every release is kept. Similarly, test management runs may retain releases.

You can use the REST API to find out what is retaining the build. And you can use the API to clear the "retain" flag regardless of dependencies. After a "retain" is cleared on an old build, the normal retention policy check cycle should delete the build.

Upvotes: -1

bryanbcook
bryanbcook

Reputation: 18328

The policies that you've linked are the retention settings for how long to keep published build artifacts within Azure DevOps not the build agents themselves. For example, if you navigate to the pipeline execution history for a pipeline and your pipeline publishes a build-artifact -- the system will keep the artifacts for 30 days for pipelines that were run on the default branch. After 30 days, the pipeline run is expunged from history. You can override this policy at each pipeline run and opt to "retain" the artifacts.

The retain build setting of the release pipeline marks artifacts produced by the linked build to be retained. This ensures that your build artifact isn't included in the build artifact retention policies if it's associated to a release that is being retained. This setting requires a Project Administrator setting to change.

Related to your scenario and disk space consumption. It's helpful to understand that the Microsoft provided cloud-hosted agents are recycled after every job. This helps to create a much more deterministic build: your pipeline builds what's defined in source and is free from side-effects from previous builds.

Azure DevOps provides several different strategies to help you emulate this behavior:

If you're running your self-hosted agents on dedicated virtual machines, you should adopt a practice of recycling the agents frequently (at least weekly) to prevent common issues related to side-effects and maintenance issues, like disk-space availability. For example:

  • Create a scheduled pipeline that runs in the Microsoft provided cloud-hosted agent pool that uses the REST API to take the agents offline so that new builds are not queued.
  • Setup a cron job or scheduled task on the agent to perform any necessary clean-up activities. At the end of the job, reboot the machine which will re-enable the agent on start-up.

Upvotes: 2

Related Questions