Executing git commands inside a build job in Visual Studio Team Services (was VSO)

Question

[Environment: Team Services, GIT, hosted build agent]

I'd like to create a Team Services build definition that is able to do the following:

1. Executing a script to generate some new files based on existing files in the repo

2. Commit/push those generated files back to repo

I can do #1 with no problem. But I'm not sure how I can do #2.

I discovered I was actually able to run git.exe from inside a build job. But I'm not sure how I can pass the credential to git. Based on the build logs, it's failing because it's trying to get the username from stdin.

I tried adding a step in the build definition with something like "git config --global user.name xxxx" but it still didn't help.

Is this a supported scenario at all? Any suggestions?

Thanks!

---

Edit

I tried the following approach in my build script:

git -c http.extraheader="AUTHORIZATION: bearer %SYSTEM_ACCESSTOKEN%" pull ...

It seemed to work for commands like pull, etc. But when I was trying to push the changes, I got the below error:

fatal: unable to access 'https://example.visualstudio.com/SampleTeam/_git/SampleRepo/': SSL read: error:00000000:lib(0):func(0):reason(0), errno 10054

Thoughts?

Answer 1

I was having this same issue. The solution was to put the git config options within the script portion of the yaml. See this GitHub issue for examples:

https://github.com/Microsoft/azure-pipelines-agent/issues/1925

Answer 2

Visual Studio Team Services (VSTS) now has built in functionality to do this:

1. Grant the account Project Collection Build Service (account_name) access to the appropriate repository in VSTS.
2. In the Agent phase, check the box to Allow scripts to access OAuth token.
3. Now within the task you can reference the variable SYSTEM_ACCESSTOKEN to access the git repository: git clone https://randomusername:${SYSTEM_ACCESSTOKEN}@instance.visualstudio.com/proj1/_git/repo

Ref: https://github.com/Microsoft/vsts-tasks/issues/962

Answer 3

This is just a followup of Tony's Blues answer.

Sorry I can't post links since my reputation is below 10, but all are placed at visualstudio website, so I'm sure you can figure this out yourself.

To allow GIT contributions within a script you need to

1. Make sure you have all stuff mentioned in VSTS Agent prerequisites done

   • Mac OS: https://www.visualstudio.com/en-us/docs/build/admin/agents/v2-osx
   • Windows: /en-us/docs/build/admin/agents/v2-windows
   • Linux: /en-us/docs/build/admin/agents/v2-linux

2. Make sure you followed instructions at /en-us/docs/build/scripts/git-commands

   • Especially add required permissions to Project Build Service account on your repository - at least Contribute (feel free to consider other permissions according to your needs) - this is the real cause of "SSL read error"

What's different between this post and Tony's one is that in our configuration (TFS 2015; VSTS Agent installed on Mac OS Sierra) we've had to add permission "Contribute" for account "Project Build Service" - so not the account with the word "collection" mentioned in name. Also be careful and not mix it up with the group named Project Collection Build Service Accounts - I believe it may be used under certain conditions but it doesn't work by default. I'm pointing this out since this is what I've accidentally did and so I've spent additional time debugging what's wrong.

Please check following picture It can be found under your project --> Control Panel --> Version control --> GIT repository

Also please be careful with system requirements since in my case (on MacOS Sierra) the part with symbolic links for two specific directories turned critical. Specific system requirements for OSX are placed at [github]/Microsoft/vsts-agent/blob/master/docs/start/envosx.md and states

Install openssl

$ brew update
$ brew install openssl

Create symbolic links to openssl libs -- this is required on MacOS (Sierra)

$ mkdir -p /usr/local/lib/
$ ln -s /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib /usr/local/lib/
$ ln -s /usr/local/opt/openssl/lib/libssl.1.0.0.dylib /usr/local/lib/

Find out your version of GIT

$ git --version

Update GIT in case you have lower than 2.9.0

$ brew update
$ brew install git

Answer 4

Sorry to answer my own question here...

I just got some hint from some TFS expert, who pointed me to this article: https://www.visualstudio.com/en-us/docs/build/scripts/git-commands, which perfectly solved my problem.

I think I should share it out to help whoever might run into the same situation as I did.

Here I am quoting the key steps (reformatted a bit):

Grant version control permissions to the build service

Go to the Version Control control panel tab

• Team Services: https://{your-account}.visualstudio.com/DefaultCollection/{your-team-project}/_admin/_versioncontrol

• On-premises: https://{your-server}:8080/tfs/DefaultCollection/{your-team-project}/_admin/_versioncontrol

On the Version Control tab, select the repository in which you want to run Git commands, and then select Project Collection Build Service (account_name). Grant permissions needed for the Git commands you want to run. Typically you'll want to grant:

• Branch creation: Allow
• Contribute: Allow
• Read: Inherited allow
• Tag creation: Inherited allow

When you're done granting the permissions, make sure to click Save changes.

Enable your build definition to run Git.exe

• On the variables tab set this variable: system.prefergit = true
• On the options tab select Allow scripts to access OAuth token.

With these settings, there is no need to install the Git Build Tools extension or tweak the Credential Manager. You don't need to explicitly set the extra header for OAuth token, either. I feel it's indeed a very neat solution. :)

But really appreciate the help from Eddie and VonC!

Answer 5

You can install Git Build Tools extension and then add "Allow Git remote access" task in your build definition. Make sure "Allow Scripts to Access OAuth Token" feature under "Options" tab is enabled.

Enable Git Remote Access

Certain operations require access to the remote repository from during a build. This task updates a remote of the Git repository on the agent to allow access to the upstream repository on Visual Studio Team Services.

Requirements

For this build task to work it is required that the Allow Scripts to Access OAuth Token option is set in the build definition options.

Parameters

Enable Git Remote Access

Remote name: Name of the remote which should be updated. Default is origin.

Related Tasks

Restore Git Remote should be called at the end of the build definition to restore the remote to its original value.

Known issues

Git-Lfs operations, like git lfs fetch still won't work with this. See this Git-Lfs issue

Add the steps for using the powershell script in the extension:

1. Create a power-shell script with the code in the "EnableGitRemoteAccess.ps1" script and add the script into source control.
2. Enable the "Allow Scripts to Access OAuth Token" option in the build definition.
3. Add a PowerShell task in build definition and set the script path the script to enable the git remote access.
4. Add another PowerShell task in build definition to commit and push the changes.

The code I use to commit and push changes:

git add .
git commit -m "changesinbuild"
git push origin master 2>&1 | Write-Host

Answer 6

Any file that you can generate from the source is generally considered as build artifact, and not added/committed/pushed to a git repo.

That being said, if you can, you should use an ssh url instead of an https one: ssh would require an ssh key, and if your private ssh key is passphrase-less, git won't have to query anything on stdin.

Another way is to use the Microsoft GCH (Git Credential Helper), which is included in Git for Windows (since Git 2.7.3, March 2016).
See this answer for an example. That would cache your login/password within the Windows Credential store.