Nomad fails to start docker containers on macOs with lima

Question

I am trying to get nomad to run on my mac with lima.

• Lima seems to work just fine :

I created a "docker-rootful" vm using the provided template :

limactl start template://docker-rootful

docker context create lima-docker-rootful --docker "host=unix:///Users/antonin/.lima/docker-rootful/sock/docker.sock"
docker context use lima-docker-rootful

I can run and portforward to my host containers using docker with lima.

docker run --rm -p8080:80 nginx 

And nginx is port forwarded to my mac host on localhost:8080.

• I also configured nomad docker plugin to use the correct containerd socket :

plugin "docker" {
    config {
      // endpoint = "unix:///var/run/docker.sock"
      // use lima
      endpoint = "unix:///Users/antonin/.lima/docker-rootful/sock/docker.sock"

• But nomad dev setup seems to be incorrect

I tried the sample application from nomad (a redis cache in a container), nomad agent fails to start the container. I guess there is too much network indirection ...

Here is my command to start nomad (according to nomad FAQ) :

sudo nomad agent -dev -config=docker.nomad -bind=0.0.0.0 -network-interface=en0

Here is the message logged by the nomad agent :

2023-02-15T22:57:24.796+0100 [DEBUG] client.driver_mgr.docker: failed to start container: driver=docker container_id=196030610c2795d6876386e799e03dd9f9afa3b391d6c093ab1d03616daf1949 attempt=5 error="API error (500): driver failed programming external connectivity on endpoint redis-bc2b6d1d-9241-ccb8-c5ad-3e649f018208 (df95904fed93d4be70fc0c3bf1682dbead84ba71f2a321443fba09ff0c43dd69): Error starting userland proxy: listen tcp4 192.168.1.120:28033: bind: cannot assign requested address"

192.168.1.120 is my host ip on my LAN.

additional information :

• Here is the nomad job spec :

job "example" {
  type = "service"
  update {
    max_parallel = 1
    min_healthy_time = "10s"
    healthy_deadline = "3m"
    progress_deadline = "10m"
    auto_revert = false
    canary = 0
  }
  migrate {
    max_parallel = 1
    health_check = "checks"
    min_healthy_time = "10s"
    healthy_deadline = "5m"
  }
  group "cache" {
    count = 1
    network {
      port "db" {
        to = 6379
      }
    }
    service {
      name     = "redis-cache"
      tags     = ["global", "cache"]
      port     = "db"
      provider = "nomad"
      check {
        name     = "alive"
        type     = "tcp"
        interval = "10s"
        timeout  = "2s"
      }
    }
    restart {
      interval = "30m"
      delay = "15s"
      mode = "fail"
    }
    ephemeral_disk {
      size = 300
    }

    task "redis" {
      driver = "docker"
      config {
        image = "redis:7"
        ports = ["db"]
        auth_soft_fail = true
      }
      resources {
        cpu    = 200 # 500 MHz
        memory = 100 # 256MB
      }
    }
  }
}

• Here is the output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.0)
  compose: Docker Compose (Docker Inc., v2.15.1)
  dev: Docker Dev Environments (Docker Inc., v0.0.5)
  extension: Manages Docker extensions (Docker Inc., v0.2.17)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.23.0)

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 20.10.22
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 3.841GiB
 Name: docker-desktop
 ID: 3FQL:QB3Z:4XQH:7W3B:7D5J:KOTR:LADV:MXVC:UUCI:BDAN:45IT:NMDD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Username: antonin42
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false