Build and Test iOS app with SPM on GitHub Actions

Question

I have been trying to build and then run unit tests for my iOS app that has some dependencies managed via SPM. It works fine, however, I am trying to speed up the process by caching dependencies, so I don't have to re-download or re-compile them until there is change, which is where I am failing.

I have tried tens of different configurations and at this point I must say I got lost so badly that I am not even sure what I am trying to achieve is possible.

Here is the relevant part of the jobs: section of my .yaml workflow:

build:
  needs: comment-on-pr-acknowledge
  runs-on: macos-14 # self-hosted
  steps:
  - name: Checkout
    uses: actions/checkout@v4
    with:
      fetch-depth: 0
     
  # The following code is from: https://github.com/actions/cache/blob/main/examples.md#swift---swift-package-manager
  # Please note, no matter what I try (different path and key pairs) I always get the following printed out:
  # "Cache not found for input keys: macOS-spm-<redacted_hash>, macOS-spm-"
  - uses: actions/cache@v4
    with:
    path: .build
    key: ${{ runner.os }}-spm-${{ hashFiles('**/Package.resolved') }}
    restore-keys: |
      ${{ runner.os }}-spm-
          
  - name: Build
    run: |
      xcodebuild clean \
          -target "MyAppName" \
          -scheme "MyAppName-Staging" \
          -destination "OS=17.4,name=iPhone 15 Pro,platform=iOS Simulator" \
          -derivedDataPath "DerivedData" build-for-testing | xcpretty --color --simple

  - name: Test
    run: |
     xcodebuild test-without-building \
          -scheme "MyAppName-Staging" \
          -destination "OS=17.4,name=iPhone 15 Pro,platform=iOS Simulator" \
          -derivedDataPath "DerivedData" \
          -enableCodeCoverage YES \
          -only-testing:Tests | xcpretty --color --simple

The whole thing takes 15-20 minutes which is unacceptable because it only takes ~30 seconds when I build the project on my local machine and a few more seconds on top of that for unit tests.

Some notes:

• When the build step starts, nothing gets printed for about ~90 seconds and then it starts printing things. When I run the exact command on my local terminal, it kicks off immediately. It feels like something causing hanging when running through Actions but I can't figure out what could possibly hang (it's also weird that it does not kick off immediately when it's running on my local machine when triggered by GitHub Actions i.e., self-hosted job)
• The build process usually takes 6-7 minutes but unit test always takes a few minutes longer which I find very weird because test-without-building command supposedly only runs the tests without any additional computation
• I have tried with the following Xcode versions: 15.0, 15.2.0, 15.3.0
• I have tried different approaches like xcodebuild clean test, xcodebuild test, xcodebuild build etc.

It is crucial this whole build & test process is fast as I am planning on running it for every PR (even multiple times for some).

How should my workflow look like so the project dependencies are cached across workflow runs?

I am extremely frustrated with this, any help much appreciated!