docker build not using cached layers when using --cache-from

Question

I have this following Dockerfile.

FROM ruby:3.1.3 AS build

ENV BUNDLER_VERSION=2.3.26
RUN apt-get update && \
    apt-get install -y --no-install-recommends build-essential libpq-dev  npm && \
    curl -fsSL https://deb.nodesource.com/setup_14.x | bash - && \
    apt-get install -y nodejs && \
    npm install -g n && \
    n 16.17.1 && \
    npm install -g yarn

RUN mkdir /test-app
WORKDIR /test-app

COPY ./Gemfile Gemfile.lock /test-app/
RUN gem install bundler:2.3.26
RUN bundle config build.nokogiri --use-system-libraries
RUN bundle check || bundle install
COPY ./package.json yarn.lock .npmrc /test-app/

COPY . ./

RUN bundle exec rails assets:precompile

FROM ruby:3.1.3 AS base

RUN apt-get update && \
    apt-get install -y --no-install-recommends build-essential libpq-dev  npm && \
    curl -fsSL https://deb.nodesource.com/setup_14.x | bash - && \
    apt-get install -y nodejs && \
    npm install -g n && \
    n 16.17.1 && \
    npm install -g yarn

RUN mkdir /test-app
WORKDIR /test-app

COPY --from=build /test-app ./
COPY --from=build /usr/local/bundle /usr/local/bundle

COPY entrypoint.sh /usr/bin/entrypoint.sh
RUN chmod +x /usr/bin/entrypoint.sh
EXPOSE 3000

I build the image in semaphoreci using the following docker build commands

docker pull "gcr.io/test-app/test-app-img:active";
DOCKER_BUILDKIT=1 docker build --cache-from "gcr.io/test-app/test-app-img:active" -t "gcr.io/test-app/test-app-img:$SEMAPHORE_GIT_SHA" --build-arg BUILDKIT_INLINE_CACHE=1 .;

It's not using any cached layers. I would expect it to use the cache layer at least for the apt-get since that pretty much never changes. This use to work at some point. But I cannot go back that far back in semaphore to see what changed.

Not sure how to debug why its not using the cached layers. Any thoughts?