Azure DevOps Agent Pool with scaleset waiting for agent version to upgrade before running builds

Question

I have an Azure DevOps agent pool configured to use a scale set, I also have an orchestration pipeline that triggers dozens of builds that will run on the agents provisioned by the scale set.

When builds are queued the scale set starts to provision new VMs as expected (albeit a bit slowly) and eventually the new VMs appear in the pool settings displaying having Agent version 2.165.2, however they will stay idle and not run any of the builds until they upgrade to a later version of the agent, i.e. 2.213.2.

Sometimes this upgrade takes less than a minute but in same cases it can take 15/20 mins or more, so I end up having a build queue waiting for agents that are already available but not taking any new builds until their agent version is upgraded, and during all this I am paying MS for the privilege of running idle VMs.

Is there a way for the VMs to be provisioned with the latest agent version without having to wait for this to be done as a follow up step?

Answer 1

Turns out that the behaviour we were seeing was due to us unknowingly having an older version of the agent installed on the base image, we have now updated the VM with the latest version and the scale-set behaves more responsively.

Answer 2

It is recommended that you upgrade these agents to the latest version first at an available time before using them.