Github Action Docker Build and Push not working

Question

I am trying to build a simple Dockerfile using build and push but no matter what I do it gives me permission errors. Here is the Github Action Yaml (we are running on a large runner since it is a large image):

name: Build Image and Push to ACR
on:
  pull_request:
  workflow_dispatch:
jobs:
  build-and-deploy:
    runs-on: ubuntu-latest-4-cores
    steps:
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v2
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      - name: "Azure Login using AZURE_CREDENTIALS Secret"
        uses: azure/login@v1.1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
      - name: "Docker login"
        run: |
          az account set --subscription ${{ secrets.SUBSCRIPTION_ID}}
          docker login crappdev.azurecr.io -u ${{ secrets.ACR_USERNAME }} -p ${{ secrets.ACR_PASSWORD }}
      - name: Prepare Key
        uses: webfactory/ssh-agent@v0.7.0
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
      - name: "Build and push image"
        uses: docker/build-push-action@v4
        with:
          file: ./Dockerfile
          push: true
          tags: crappdev.azurecr.io/mycompany/myimage:0.0.${{ github.run_number }},crappdev.azurecr.io/mycompany/myimage:latest
          build-args: |
            TAG=${{ vars.TAG }}
            SHA=${{ vars.SHA }}
          ssh: |
            default=${{ env.SSH_AUTH_SOCK }}

I have all the variables defined. I added a repository key and added the private key as a secret. I tested the key manually to make sure I can check out this (and only this) repository with git. I already used ssh-keyscan to add Github to the known_hosts in myuser's .ssh directory

Here is the Dockerfile:

ARG TAG
FROM crappdev.azurecr.io/mycompany/mybase:${TAG}

USER myuser
WORKDIR /home/myuser
RUN --mount=type=ssh,id=default cd /opt/envs/myvenv/ && \
   pip install git+ssh://git@github.com/mycompany/myrepo.git@${SHA} 


ENTRYPOINT ["/usr/bin/tini", "--"]

Most of the heavy lifting is in the base image. For this repo I just want to add it to the virtual-env which is defined for my user in base image. Its important to pip install it as source, so it makes use of some patched versions of libraries in the base (mostly NVidia's own torch from their Torch image).

When I run the CI on Github I get the following error:

 > [stage-0 3/3] RUN --mount=type=ssh,id=default cd /opt/envs/myenv/ &&    pip install git+ssh://git@github.com/mycompany/myrepo.git@$***SHA***:
#6 3.262 Collecting git+ssh://****@github.com/mycompany/myrepo.git@
#6 3.263   Cloning ssh://****@github.com/mycompany/myrepo.git to /tmp/pip-req-build-ilthmk_v
#6 3.263   Running command git clone -q 'ssh://****@github.com/mycompany/myrepo.git' /tmp/pip-req-build-ilthmk_v
#6 3.335   Warning: Permanently added the ECDSA host key for IP address '123.45.678.9' to the list of known hosts.
#6 3.385   git@github.com: Permission denied (publickey).
#6 3.386   fatal: Could not read from remote repository.
#6 3.386 
#6 3.386   Please make sure you have the correct access rights
#6 3.386   and the repository exists.

As noted I manually tested the ssh key on this repo, so I know 100% it can git clone it.

This line worries me a bit: #6 3.262 Collecting git+ssh://****@github.com/mycompany/myrepo.git@ Why is it empty after the @ since the SHA variable is defined, or is that just Github being secure?

Is the problem I am running as myuser not root?

Something else I am missing here?