CODEWITHSUNDEEP
dockerdocker-swarm
Andrew
Andrew

Reputation: 8674

NFS volume created manually mounts but shows empty contents

server: docker ubuntu, 18.06.3-ce local : docker for mac, 19.03.13

I have created a volume in the swarm manually, to a remote nfs server. When I try to mount this volume in a service it appears to work, but the contents are empty and any writes seem to succeed (calling code doesn't crash), but the bytes are gone. Maybe even to /dev/null.

When I declare a similar volume inside the compose file it works. The only difference I can find is the label "com.docker.stack.namespace".

docker volume create --driver local \
    --opt type=nfs \
    --opt o=addr=10.0.1.100 \
    --opt device=:/data/ \
    my_nfs

version: "3.5"

services:
  my-api:
    volumes:
      - "compose_nfs:/data1/"  # works fine
      - "externl_nfs:/data2/"  # empty contents, forgotten writes

volumes:
  externl_nfs: 
    external: true
  compose_nfs:
    driver: local
    driver_opts: 
      type: nfs
      o: addr=10.0.1.100
      device: ":/data/"

When inspecting the networks they are identical, except for that label.

{
    "CreatedAt": "2020-20-20T20:20:20Z",
    "Driver": "local",
    "Labels": {
        # label missing on the manually created one
        "com.docker.stack.namespace": "stackie"
    },
    "Mountpoint": "/var/lib/docker/volumes/externl_nfs/_data",
    "Name": "compose_nfs",
    "Options": {
        "device": ":/data/",
        "o": "addr=10.0.1.100",
        "type": "nfs"
    },
    "Scope": "local"
}

Upvotes: 0

Views: 1332

Answers (2)

BMitch
BMitch

Reputation: 263469

If you use an external volume, swarm is deferring the creation of that volume to you. Volumes are also local to the node they are created on, so you must create that volume on every node where swarm could schedule this job. For this reason, many will delegate the volume creation to swarm mode itself and put the definition in the compose file. So in your example, before scheduling the service, on each node run:

docker volume create --driver local \
    --opt type=nfs \
    --opt o=addr=10.0.1.100 \
    --opt device=:/data/ \
    external_nfs

Otherwise, when the service gets scheduled on a node without the volume defined, it appears that swarm will create the container, and that create command generates a default named volume, storing the contents on that local node (I could also see swarm failing to schedule the service because of a missing volume, but your example shows otherwise).

Upvotes: 1

Andrew
Andrew

Reputation: 8674

Answering this, since it is an older version of docker and probably not relevant to most people, considering the NFS part.

It appears to be a bug of some sort in docker/swarm.

  1. Create a NFS volume on the swarm (via api, from remote)
  2. Volume is correct on the manager node which was contacted
  3. Volume is missing the options on all other worker nodes

As some strange side effect, the volume seems to work. It can be mounted, writes succeed without issue, but all bytes written disappear. Reads work but every file is "not found", which is logical considering the writes disappearing.

On manager:

> docker network inspect external_nfs
[{
    "CreatedAt": "2020-11-03T15:56:44+01:00",
    "Driver": "local",
    "Labels": {},
    "Mountpoint": "/var/lib/docker/volumes/externl_nfs/_data",
    "Name": "externl_nfs",
    "Options": {
        "device": ":/data/",
        "o": "addr=10.0.1.100",
        "type": "nfs"
    },
    "Scope": "local"
}]

On worker:

> docker network inspect external_nfs
[{
    "CreatedAt": "2020-11-03T16:22:16+01:00",
    "Driver": "local",
    "Labels": {},
    "Mountpoint": "/var/lib/docker/volumes/externl_nfs/_data",
    "Name": "externl_nfs",
    "Options": null,
    "Scope": "local"
}]

Upvotes: 0

Related Questions