Builds fail due to timeout during Get Sources step in TFS 2018

Question

Our TFS build agents are having seemingly random issues with the Get Sources step where the git fetch command hangs and times out after an hour.

The last portion of the logs shows it got everything it needed but decided to just stop until it timed out.

2018-04-12T10:00:11.9145204Z Resolving deltas:  97% (123/126)   
2018-04-12T10:00:11.9145204Z Resolving deltas:  99% (125/126)   
2018-04-12T10:00:11.9145204Z Resolving deltas: 100% (126/126)   
2018-04-12T10:00:11.9145204Z Resolving deltas: 100% (126/126), completed with 17 local objects.
2018-04-12T11:00:03.8383793Z ##[debug]Re-evaluate condition on job cancellation for step: 'Get Sources'.
2018-04-12T11:00:04.1196536Z ##[error]The operation was canceled.
2018-04-12T11:00:04.1352781Z ##[debug]System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.VisualStudio.Services.Agent.ProcessInvoker.<ExecuteAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.GitCommandManager.<ExecuteGitCommandAsync>d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.GitCommandManager.<GitFetch>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.GitSourceProvider.<GetSourceAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildJobExtension.<GetSourceAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.JobExtensionRunner.<RunAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.<RunStepAsync>d__1.MoveNext()
2018-04-12T11:00:04.1352781Z ##[section]Finishing: Get Sources

We have 4 build agents setup and usually one or two have this problem each night during builds.

When I look into the agent's directory and run a git status command, I get a message saying:

Rename from 'C:\BuildAgent1\_work\1\s\.git\index.lock' to C:\BuildAgent1\_work\1\s\.git\index' failed. Should I try again? (y/n)

Entering 'y' just causes that message to repeat. Looking into the .git/ folder, there is no index.lock file present.

Issuing another build manually and the build gets past the Get Sources step without issue.

Why are the builds hanging after fetching everything?

Answer 1

So, the solution for this issue is cleaning the sources directory.

• Set Clean to true and Clean options select Sources directory under Get Sources step.
• Alternatively you can delete the source folder manually on agent machine. (e.g.: C:\Agent\_work\2\s\, basically delete the .git folder will resolve the issue C:\Agent\_work\2\s\.git)