CODEWITHSUNDEEP
pythonazure-devops
JitterRobin
JitterRobin

Reputation: 31

Azure Devops Pipelines with clean Python environment

I have an Azure Devops CI build set up to run a set of tests on my Python codebase. The pipeline runs using a local PC sitting on my desk, which I have added to the build pool. I installed Python 3.6 in the _work\_tool folder of the build agent and I start my pipelines .yaml files with a task:

- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.6'

which grabs Python 3.6 and uses it for all "python" commands in subsequent steps.

This all works fine, but in order to run my tests, the packages in my python repo have to be installed on the python environment. I do this by calling a custom script that manually installs the packages one by one (essentially calls python setup.py install on each, which pulls in external dependencies as well). The problem is that I'm not convinced this installs the packages to a clean environment each time and so the python installation can end up using packages from previous build jobs.

Is there a way to ensure that each time a pipeline runs, the build agent uses a completely clean Python virtual environment for it?

Upvotes: 1

Views: 4571

Answers (1)

Hugh Lin
Hugh Lin

Reputation: 19361

In yaml, when you run a pipeline on a self-hosted agent, by default, none of the sub-directories are cleaned in between two consecutive runs. As a result, you can do incremental builds and deployments, provided that tasks are implemented to make use of that. You can override this behavior using the workspace setting on the job. Refer to this document for details.

- job: myJob
  workspace:
    clean: outputs | resources | all # what to clean up before the job runs
  • outputs: Delete Build.BinariesDirectory before running a new job.
  • resources: Delete Build.SourcesDirectory before running a new job.
  • all: Delete the entire Pipeline.Workspace directory before running a new job.

Note : $(Build.ArtifactStagingDirectory) and $(Common.TestResultsDirectory) are always deleted and recreated prior to every build regardless of any of these settings.

As you said in the question:"I do this by calling a custom script that manually installs the packages one by one". If your packages are not installed in the workspace directory, the clean option cannot delete these packages.

enter image description here

So, as a workaround, you can empty the folder where the package is installed via cmd or script. For example:

rmdir /s/q path\folderName
mkdir path\folderName

Upvotes: 2

Related Questions