CODEWITHSUNDEEP
postgresqldockerdocker-composedocker-volume
Alex Lenail
Alex Lenail

Reputation: 14410

How to persist data in a dockerized postgres database using volumes

My docker compose file has three containers, web, nginx, and postgres. Postgres looks like this:

postgres:
  container_name: postgres
  restart: always
  image: postgres:latest
  volumes:
    - ./database:/var/lib/postgresql
  ports:
    - 5432:5432

My goal is to mount a volume which corresponds to a local folder called ./database inside the postgres container as /var/lib/postgres. When I start these containers and insert data into postgres, I verify that /var/lib/postgres/data/base/ is full of the data I'm adding (in the postgres container), but in my local system, ./database only gets a data folder in it, i.e. ./database/data is created, but it's empty. Why?

Notes:

UPDATE 1

Per Nick's suggestion, I did a docker inspect and found:

    "Mounts": [
        {
            "Source": "/Users/alex/Documents/MyApp/database",
            "Destination": "/var/lib/postgresql",
            "Mode": "rw",
            "RW": true,
            "Propagation": "rprivate"
        },
        {
            "Name": "e5bf22471215db058127109053e72e0a423d97b05a2afb4824b411322efd2c35",
            "Source": "/var/lib/docker/volumes/e5bf22471215db058127109053e72e0a423d97b05a2afb4824b411322efd2c35/_data",
            "Destination": "/var/lib/postgresql/data",
            "Driver": "local",
            "Mode": "",
            "RW": true,
            "Propagation": ""
        }
    ],

Which makes it seem like the data is being stolen by another volume I didn't code myself. Not sure why that is. Is the postgres image creating that volume for me? If so, is there some way to use that volume instead of the volume I'm mounting when I restart? Otherwise, is there a good way of disabling that other volume and using my own, ./database?

Upvotes: 423

Views: 445786

Answers (6)

uceumern
uceumern

Reputation: 937

I had a similar issue where postgres would create an anonymous volume in addition to the specified one.

  postgres:
    container_name: postgres
    image: postgres:15
    environment:
      PGUSER: postgres
      POSTGRES_PASSWORD: postgres
      PGDATA: /data/postgres
    volumes:
      - postgres_data:/data/postgres

Turns out that specifying PGDATA environment variable causes that. The container would properly use the postgres_data volume, but would in addition create an empty anonymous volume.

You can find out the used volumes by using docker inspect postgres, here is the Mounts section of the output:

    "Mounts": [
        {
            "Type": "volume",
            "Name": "modular-service-layer_postgres_data",
            "Source": "/var/lib/docker/volumes/modular-service-layer_postgres_data/_data",
            "Destination": "/data/postgres",
            "Driver": "local",
            "Mode": "z",
            "RW": true,
            "Propagation": ""
        },
        {
            "Type": "volume",
            "Name": "ceef11ea50a07400a798fbda75db4896f35a4a33d41c70cc68f976c187736dbd",
            "Source": "/var/lib/docker/volumes/ceef11ea50a07400a798fbda75db4896f35a4a33d41c70cc68f976c187736dbd/_data",
            "Destination": "/var/lib/postgresql/data",
            "Driver": "local",
            "Mode": "",
            "RW": true,
            "Propagation": ""
        }
    ],

I fixed that by using the proper mount point directly without setting PGDATA:

  postgres:
    container_name: postgres
    image: postgres:15
    environment:
      PGUSER: postgres
      POSTGRES_PASSWORD: postgres
    volumes:
      - postgres_data:/var/lib/postgresql/data

Upvotes: 1

David Maze
David Maze

Reputation: 158647

The postgres image's Dockerfile contains this line:

VOLUME /var/lib/postgresql/data

When a container starts up based on this image, if nothing else is mounted there, Docker will create an anonymous volume and automatically mount it. You can see this volume using commands like docker volume ls, and that's also the second mount in the docker inspect output you quote.

The main consequence of this is that your external volume mount must be on /var/lib/postgresql/data and not a parent directory. (Also see @AlexLenail's answer.)

If you mount a host directory on /var/lib/postgresql instead, then:

  1. Docker internally sorts the mounts, so parent directories get mounted first.
  2. The host directory is mounted on /var/lib/postgresql.
  3. Nothing is mounted on /var/lib/postgresql/data, so Docker creates the anonymous volume.
  4. The mount point does not exist (it is in the empty bind-mounted host directory) so Docker creates it, also creating the ./database/data directory on the host.
  5. The anonymous volume is mounted on that directory, but only in the container filesystem.

The result of this is what you see, the database apparently operates correctly but its data is not necessarily persisted and you get only the empty data directory on the host.

Upvotes: 22

Nishchit
Nishchit

Reputation: 19074

You can create a common volume for all Postgres data

docker volume create pgdata

or you can set it to the compose file

version: "3"
services:
  db:
    image: postgres
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgress
      - POSTGRES_DB=postgres
    ports:
      - "5433:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data
    networks:
      - suruse
volumes:
  pgdata:

It will create volume name pgdata and mount this volume to container's path.

You can inspect this volume

docker volume inspect pgdata

// output will be
[
    {
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/pgdata/_data",
        "Name": "pgdata",
        "Options": {},
        "Scope": "local"
    }
]

Upvotes: 213

Alex Lenail
Alex Lenail

Reputation: 14410

Strangely enough, the solution ended up being to change

volumes:
  - ./postgres-data:/var/lib/postgresql

to 

volumes:
  - ./postgres-data:/var/lib/postgresql/data

Upvotes: 538

leldo
leldo

Reputation: 386

I think you just need to create your volume outside docker first with a docker create -v /location --name and then reuse it.

And by the time I used to use docker a lot, it wasn't possible to use a static docker volume with dockerfile definition so my suggestion is to try the command line (eventually with a script ) .

Upvotes: 3

Nick Burke
Nick Burke

Reputation: 969

I would avoid using a relative path. Remember that docker is a daemon/client relationship.

When you are executing the compose, it's essentially just breaking down into various docker client commands, which are then passed to the daemon. That ./database is then relative to the daemon, not the client.

Now, the docker dev team has some back and forth on this issue, but the bottom line is it can have some unexpected results.

In short, don't use a relative path, use an absolute path.

Upvotes: 19

Related Questions