Invoke NVidia runtime failed with docker compose on AGX Orin

Question

I'm deploying CI flow of my CUDA project on AGX Orin with Gitlab-runner.

After I built my docker image, I ran the container manually with the command docker run -it --rm --runtime nvidia <image name>, and all test cases passed.

When I tried to use docker compose, only parts of my test cases passed.

Did I lost any settings?

Below is my config.toml, docker-compose.yml: config.toml

[[runners]]
  name = "ddjetson AGX Orin"
  url = "xxxxx"
  id = 404
  token = "xxxxx"
  token_obtained_at = 2024-08-16T03:20:53Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "jetsondev"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = "1g"
    network_mtu = 0
    pull_policy = "if-not-present"
    runtime = "nvidia"

docker-compose.yml

version: '3'

services:
  gitlab-runner:
    image: gitlab/gitlab-runner:latest
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    container_name: gitlab-runner
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/gitlab-runner/config.toml:/etc/gitlab-runner/config.toml