GitHub action docker build (self-hosted runner) hanging on poetry install with private repo

Question

I've deployed github-action-runnet-set helm chart to use GitHub Actions for my organization. My goal is to have the runner builds a Docker image, then push it to private registry. I installed github-action-runnet-set helm chart by this command:

INSTALLATION_NAME="arc-runner-set"
NAMESPACE="arc-runners"
GITHUB_CONFIG_URL="https://github.com/<org>"
GITHUB_APP_SECRET=<github_app_secret>
helm install "${INSTALLATION_NAME}" \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
    --set containerMode.type=dind \
    --set githubConfigSecret="${GITHUB_APP_SECRET}" \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

The runner is working properly with simple workflow file, it is able to log into private registry, but it fails on docker build step, for more specific, it hangs on step RUN --mount=type=ssh poetry install --no-interaction.

This is my workflow file.

jobs:
  build-and-push:
    runs-on: arc-runner-set

    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-qemu-action@v3
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          username: ${{ secrets.REGISTRY_USERNAME }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
          registry: ${{ secrets.REGISTRY_SERVER }}
      - run: echo "${{ secrets.SSH_PRIVATE_KEY }}" > ssh-private-key
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ secrets.REGISTRY_SERVER }}/docker/user-service:${{ github.ref_name }}.${{ github.sha }}
          ssh: |
            default=ssh-private-key

This is my pyproject.toml file:

[tool.poetry]
...

[tool.poetry.dependencies]
python = "^3.12"
...
core = { git = "git@github.com:<org>/core.git" }
...

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

and this is my Dockerfile:

FROM python:3.12

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

WORKDIR /app
RUN python -m pip install poetry
RUN poetry config virtualenvs.create false

COPY pyproject.toml /app

RUN --mount=type=ssh mkdir -p ~/.ssh
RUN --mount=type=ssh ssh-keyscan github.com > ~/.ssh/known_hosts
RUN --mount=type=ssh poetry install --no-interaction

CMD ["python", "app.py"]

It stucks or hanging at #12 41.63 - Installing core (0.1.1 8666dad) for hours in GitHub actions.

I can't be sure where problem is (my ssh key, my hosted runner or my private repo, ...) I did try to change runs-on to ubuntu-latest to use Github-hosted runner and it works.

I really appreciate everyone helps