Proper Workflow for Using GitHub submodules with Azure Devops Pipelines (Adding and Checking out)

Question

I know about this question: Checkout git submodule from azure pipeline but this answer does not get into how to properly create the submodule in the first place using GitHub PATs.

[My setup]

• I have an Azure Devops pipeline and associated Azure Devops git repo holding the YML script and the "parent" git repository.
• The parent repository is to host several git submodules from PRIVATE GitHub repos.
• I am using PATs for GitHub repo authentication.
• I have generated a PAT, prefixed it, base64 encoded it and saved it into an Azure Key Vault for later retrieval by the pipeline. Retrieval tested and works.

[My Next Task] - Need to ADD a submodule to the Azure Devops project's "parent" git repo. This is where I am failing to find any info. There is lots of info on performing the later stage of a submodule init and update, no info on getting the submodule registered in the first place, when PATs are involved.

[What I tried]

• manual clone of the parent repo to a linux host.
• add the submodule: git submodule add https://PAT:<raw PAT>@github.com/<account>/<sm1>.git
• now the .gitmodules file has an unwanted un-encoded PAT in it, I edit the .gitmodules file and remove the PAT:<raw PAT> part of the URL for submodule .
• I add and commit the changes back to Azure Devops.
• In the pipeline YML, as per the above link's suggestion, add a script: task to the pipeline and manually try to sync up the submodule. The pipeline already pulls the base64 encoded PAT from the vault and I place it into an environment variable:

[YML segment]

- task: AzureKeyVault@2
  inputs:
    azureSubscription: 'vault-access'
    KeyVaultName: '<KVN>'
    SecretsFilter: '<my-secret>'
    RunAsPreJob: false
    
- script: |
    echo "Updating private Github submodules..."
    git -c http.https://github.com/<account>/<sm1>.git.extraheader="AUTHORIZATION: basic $GITHUB_PKEY_ENC64" submodule update --init --recursive
  displayName: 'Submodule initialization'
  env:
    GITHUB_PKEY_ENC64: $(<my-secret>)

The above script step fails and I have determined that the git command is manually asking for keyboard input. When manually run on a fresh clone of the parent, I get this:

git -c http.https://github.com/<account>/<sm1>.git.extraheader="AUTHORIZATION: basic $GITHUB_PKEY_ENC64" submodule update --init --recursive 
Submodule '<sm1>' (https://github.com/<account>/<sm1>.git) registered for path '<sm1>'
Cloning into '/<path>/git/<parent>/<sm1>'...
Username for 'https://github.com':

Why is it asking for a username and not using the config params added as part of -c ?

Is the method by which I added the submodule in the first place the proper way to do it?

[ADDITIONAL : Mar 16 2022]

Error from the pipeline run:

Starting: Submodule initialization
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.200.2
Author       : Microsoft Corporation
Help         : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /opt/azagent/_work/_temp/3e05f5cf-5af1-4f4f-9da6-c71a5c40ed3a.sh
Updating private Github submodules...
Cloning into '/opt/azagent/_work/7/s/sm1'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
fatal: clone of 'https://github.com/account/sm1.git' into submodule path '/opt/azagent/_work/7/s/sm1' failed
Failed to clone 'sm1'. Retry scheduled
Cloning into '/opt/azagent/_work/7/s/sm1'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
fatal: clone of 'https://github.com/account/sm1.git' into submodule path '/opt/azagent/_work/7/s/sm1' failed
Failed to clone 'sm1' a second time, aborting
##[error]Bash exited with code '1'.
Finishing: Submodule initialization

My .gitmodules file:

[submodule "sm1"]
    path = sm1
    url = https://github.com/account/sm1.git

(!) I suspect that the .gitmodules file is incorrect for use as a PAT authenticated URL? Please someone confirm this.

Answer 1

If your main project and your submodules are in the same GitHub organization:

You can register your submodules by enabling the "Checkout submodules" option. As you are using YAML pipeline, you can try following steps:

1. Go to the edit page of your pipeline.
2. Click on the three dots button on the top right corner and select "Triggers".
3. Select "YAML" -> "Get sources".
4. Check the "Checkout submodules" option and select your recursion level.

The token stored in the GitHub service connection is used to access the sources. You don't need to authenticate it using PAT manually.

If your main project and your submodules are in the different GitHub organization:

You can authorize your submodule using the following command:

git -c http.https://<url of submodule repository>.extraheader="AUTHORIZATION: basic <BASE64_ENCODED_TOKEN_DESCRIBED_ABOVE>" submodule update --init --recursive

The above script step fails and I have determined that the git command is manually asking for keyboard input.

I can run this command in pipeline successfully. You can provide your error information for further investigation.