How to setup Docker Compose without a Dockerfile?

Question

I have spend the last few hours trying to setup 2 default images of nodejs 14 and rethinkdb 2.3.5, so sorry if the tone is a little frustrated but I am currently frustrated.

My requirements are seemingly super simple.

I want to:

1. download the default images for nodejs 14 and rethinkdb 2.3.5
2. copy everything from my current directory into the nodejs 14 image.
3. I want the nodejs image to depend on the RethinkDB image.
4. run 2 commands in the nodejs 14 image; npm ci and npm test.
5. see the stdout from the tests.

I do not need:

1. any ports to be accessible to the host machine.
2. to customize any Dockerfile or make any changes to the default images.
3. any updates from the host machine file-system to the containers.
4. to copy any data from the host machine to the RethinkDB container.

Why?

I want the tests to be reproducible across all developer machines - currently no CI. Regardless of where the developer has the project on their hard drive.

I have a single docker-compose.yml file.

version: "3"
services:
  tests:
    image: node:14
    ports:
      - "3000:3000"
    # command:
    #  - npm ci
    #  - npm test
    volumes:
      - ".:/cli-app"
    depends_on:
      - rethinkdb

  rethinkdb:
    image: rethinkdb
    ports:
      - "28015:28015"
      - "8080:8080"
    volumes:
      - "./data: /data"
    command: rethinkdb --bindall --data /data

Answer 1

The point of this answer is not to give the must succinct explanation (briefly and clearly expressed) as possible but to highlight all the confusion that the current documentation at docs.docker.com and hub.docker.com create. Eventually I/we will get it right and a succinct answer can be written.

The corrected docker-compose.yml:

version: "3"
services:
  tests:
    image: "node:14"
    user: "node"
    working_dir: /home/node/app
    volumes:
      - ./:/home/node/app
    container_name: nodejs
    depends_on:
      - rethinkdb
    command: bash -c "npm ci && npm test"

  rethinkdb:
    image: rethinkdb:2.3.5
    container_name: rethinkdb

The frustrating part!

Right off the bat, the documentation at docs.docker.com and hub.docker.com is arguably the worse documentation ever written, since it is a) wrong, b) assumes prior knowledge.

If any of the following is wrong - blame the horrible documentation.

No, you do not need a Dockerfile unless you plan to built your own image.

So after wasting an hour or so on different outdated examples, you might be lucky to discover that everything you have tried with context to get around the absolute path examples... does not matter at all unless you are creating your own image from scratch (which 90% of docker users, do not need).

Tip use docker system prune to delete all those unfortunate useless docker containers you've creating by following example.

Next up, find the correct docker containers.

Presenting: nodejs official docker image!

Did you see the image name that we need?

Not one place does it say image. You just have to know.

Line by line explanation to docker-compose.yml

version: "3"

Uses version 3.x of the syntax. The syntax varies from Docker engine versions as listed at Compose and Docker compatibility matrix.

services:

Each container image is a service in Docker Compose terminology. tests and rethinkdb are my names for 2 images. You can name them as you want but we will use this name later to create a dependency between the 2 images (one needs to be online before the other).

services:
  tests:
    ...
  rethinkdb:
    ...

tests and rethinkdb are the two services that we will get Docker Compose to run for us.

Fix: 1. download the default images for nodejs 14 and rethinkdb 2.3.5

    image: "node:14"

Luckily the nodejs Docker developers included extremely nice documentation in lieu of proper standard documentation at hub.docker.com/_/node.

Image variants: The defacto image node:<version> and node:<version>-alpine node:<version>-alpine image is based on the popular Alpine Linux project, which is much smaller than most distribution base images (~5MB), and thus leads to much slimmer images in general..

    image: rethinkdb:2.3.5

Unfortunately, the less funded RethinkDB project, Docker Hub page does not have such information. Actually if you scroll all the way down to the bottom of the page, you will find Image Variants (sorry, I can not link to headings on hub.docker.com since they do not have an id or name attribute). The 2 versions you will want to use is either rethinkdb:<version> or rethinkdb:<version>-slim.

Q: So what about the much more dominant Supported tags and respective Dockerfile links section?

A: They are less frequently needed and are specialized docker images. In the RethinkDB case, it is the RethinkDB database install on Debian Buster and CentOS. There is also a link to some versions but not all. So it's a selected list of images, you probably do not want. Remember you have to click on Tags or the small grey link; View Available Tags (see the image above with the red squares - sorry no anchor linking support on SO either).

Fix: 2. copy everything from my current directory into the nodejs 14 image.

    user: "node"
    working_dir: /home/node/app
    volumes:
      - ./:/home/node/app

If you look up user or working_dir on docs.docker.com, you are out of luck without prior knowledge with docker. It only states that:

Each of these is a single value, analogous to its docker run counterpart. Note that mac_address is a legacy option.

The goal here is to copy the current directory (where docker-compose.yml is located) to the tests service which pulls the node:14 image. We need to create a directory in our container where our current directory on our machine will be copied to. Later on, we want to execute some commands in that directory and we do not want to run stuff as sudo, so the natural placement is in our user home directory (~/).

We need:

1. A user in our nodejs container.
2. Create a directory in the user home directory in our nodejs container.
3. Map our current directory on our machine to the new directory in our nodejs container.

After reading How to use this image that the nodejs developers wrote, we can see in the example that there is a node user in the image. The example also shows which Docker Compose configurations we need.

    user: "node"

Presumably use the node user defined in image: "node:14".

    working_dir: /home/node/app

Create an app directory inside the node user's home directory.

    volumes:
      - ./:/home/node/app

Map our current directory on our machine to the /home/node/app directory inside the tests container (which uses image: "node:14").

Fix: 3. I want the nodejs image to depend on the RethinkDB image.

    container_name: nodejs

Turns out that defining container_name is purely cosmetic. It does not help you to link network or start one container before the other. It is just a name. While your containers are running you can use docker ps to see them or docker ps -a to see all containers, even shut downed ones. In the NAMES column the container_name is written. If you do not define container_name, then they will be called your directory name and the service name, post-fixed with a incremental number.

docker ps with container_name while running:

CONTAINER ID        IMAGE               PORTS                            NAMES
5272576f8555        node:14                                              nodejs
fb11d5ce049b        rethinkdb:2.3.5     8080/tcp, 28015/tcp, 29015/tcp   rethinkdb

docker ps without container_name while running:

CONTAINER ID        IMAGE               PORTS                            NAMES
528e5ee37956        node:14                                              data_access_layer_tests_1
e80682b806fc        rethinkdb:2.3.5     8080/tcp, 28015/tcp, 29015/tcp   data_access_layer_rethinkdb_1

    depends_on:
      - rethinkdb

depends_on is the magic one! It tells Docker Compose that the rethinkdb container must be online started before this container.

This in a rare case of the Docker Compose documentation (depends_on) actually being good.

Unfortunately depends_on does not guarantee that the depending image is online, as you would expect, but the documentation is also very clear in this case and offer another solution. In our case, it does not matter much since the rethinkdb container starts fast enough (npm ci in nodejs will take way longer than the startup time of rethinkdb).

Fix: 4. run 2 commands in the nodejs 14 image; npm ci and npm test.

    command: "npm ci && npm test"

Now we can run some commands inside our docker container, as if we where running them on our machine in our project directory. Of course not. The above will only execute npm ci. See the SO answer Using Docker-Compose, how to execute multiple commands. Docker uses an obscure difficult to fathom variant of shell script that can execute one, and only one command, with parameters. The documentation says it is similar to the docker CMD key but neither sites explain why some POSIX shell commands work and others don't but we assume that it made life easier for Docker developers :)

The correct way to execute multiple commands, is:

    command: bash -c "npm ci && npm test"

Please see "Using Docker-Compose, how to execute multiple commands".

npm ci installs all npm packages from package-lock.json and npm test runs our "test" script in package.json.

You might think that you can write multiple commands in an YAML array but doing so will, in this case apparently, tell node to require them as modules inside the container and you will get errors like:

nodejs       | internal/modules/cjs/loader.js:883
nodejs       |   throw err;
nodejs       |   ^
nodejs       |
nodejs       | Error: Cannot find module '/home/node/app/npm ci'
nodejs       |     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
nodejs       |     at Function.Module._load (internal/modules/cjs/loader.js:725:27)
nodejs       |     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
nodejs       |     at internal/main/run_main_module.js:17:47 {
nodejs       |   code: 'MODULE_NOT_FOUND',
nodejs       |   requireStack: []
nodejs       | }

    command:
      - "npm ci"
      - "npm test"

The above will not work!

Instead use the usual && to execute a series of successful commands. Again, Using Docker-Compose, how to execute multiple commands is the correct documentation.

Actually, forget the above and use entrypoint to point to a bash file where you write all of your commands.

Fix: 5. see the stdout from the tests.

DO NOT USE docker-compose up -d!

You do not have to do anything here. Even though every Docker Compose tutorial will tell you to run in detached mode. You will not see stdout or stderr. Just loose the -d.

docker-compose up is the way to go!

If you really have to run in detached mode, you can see the output via docker logs [container id] and get the container id via docker ps or docker ps -a. The latter is used if your containers are not currently running.

docker-compose.yml keys that look like they are needed

Turns out that Docker comes with a lot of sane defaults which you should not mess with unless you really have to.

    ports:
      - "28015:28015"
      - "8080:8080"

You do not need to expose ports in a docker-compose.yml unless you need to expose the ports to your local machine or outside network, even though every example out there does it. For example, if you need access to localhost:8080 (in RethinkDB that is the dashboard), then you have to add:

ports:
  - "8080:8080"

Your other services/containers will have access to both port 28015 and 8080, using the service name as hostname, without you have to specify anything in your docker-compose.yml. E.g. in this case rethinkdb:28015. See below for more information.

You do not need to expose ports in a `docker-compose.yml` even though every example out there does it. Chances are that the image you use, already exposes the default ports used by the containerized software you need. The above example is the default ports that RethinkDB uses and they are exposed without you having to write them.

---

    links:
      - rethinkdb

links seems like a way to connect two distinct containers but it turns out that all containers share network, so you do not have to. The official documentation comes with a big red WARNING, which suggest that you use user-defined networks. Which in turn says that:

By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.

That's a convoluted way of saying that inside your Docker Compose containers, you can connect to ports and IP address just as you would, had it been on your local machine.

So if container A's image exposes 172.18.0.2:28015 then you can connect from container B using that exact address. E.i. IP: 172.18.0.2 and port: 28015. Scratch that! The exposed IP addresses are not stable. Docker will change them, for various (unknown) reasons.

Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.

Means that your service name is used as hostname, in the same way you would define it in your /etc/hosts file. Or similar to how DNS links stackoverflow.com to 151.101.193.69.

So if container rethinkdb exposes port 28015 then it will be accessible from the nodejs container via rethinkdb:28015.

BEWARE that the documentation says:

version: "3.9"
services:
  web:
    build: .
    ports:
      - "8000:8000"
  db:
    image: postgres
    ports:
      - "8001:5432"

Each container can now look up the hostname web or db and get back the appropriate container’s IP address. For example, web’s application code could connect to the URL postgres://db:5432 and start using the Postgres database.

In the example for Networking in Compose postgres is the image, db is the service name and 5432 is the port number. This is does not work in my experience. You need to use the service name and never the image name. So the correct way to connect from web to db is to use the URL db:5232. @jonrsharpe points out that the protocol (postgres://) just happens to match the image name.

The following is a correction of my previous understanding of the example.

In the example for Networking in Compose postgres is the protocol (similar to https), db is the service name and 5432 is the port number. You need to use the service name, to get the correct IP address. So the correct way to connect from web to db is to use the URL [protocol]://db:5232. Where protocol can be http, https, progres, etc.

Since all containers are within the same network, you do not need the ports key unless you need to expose a service to your local machine (or outside network).

---

    volumes:
      - "./data: /data"

Every RethinkDB Dockerfile has this but this is only needed if you want to copy files from your local machine into the container. In this case we do not want to preload the database with anything, so we do not have any files to seed the database with, hence the volumes key is not needed.