Running CEPH in docker

Question

I have weak DevOps skills and I don't know much about Docker. I need to make Java friends with Ceph. To start, I'm trying to run CEPH in my docker container. I look at Intellij Idea and understand that not all containers are running. My docker-compose looks like this:

version: '2.1'

services:

  mon1:
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "mon"
    environment:
      MON_IP: ${MON1_IP}
      CEPH_PUBLIC_NETWORK: ${MON1_CEPH_PUBLIC_NETWORK}
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
    networks:
      - ceph-cluster-net

  mgr:
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "mgr"
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
    depends_on:
      - mon1
    networks:
      - ceph-cluster-net
    ports:
      - ${DASHBOARD_PORT}:7000
    expose:
      - ${DASHBOARD_PORT}

  osd1:
    pid: host
    privileged: true
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "osd"
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
      - /dev/:/dev/
    environment:
      OSD_DEVICE: ${OSD1_DEVICE}
      OSD_TYPE: disk
      OSD_FORCE_ZAP: 1
    depends_on:
      - mon1
    networks:
      - ceph-cluster-net

  osd2:
    pid: host
    privileged: true
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "osd"
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
      - /dev/:/dev/
    environment:
      OSD_DEVICE: ${OSD2_DEVICE}
      OSD_TYPE: disk
      OSD_FORCE_ZAP: 1
    depends_on:
      - mon1
    networks:
      - ceph-cluster-net

  osd3:
    pid: host
    privileged: true
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "osd"
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
      - /dev/:/dev/
    environment:
      OSD_DEVICE: ${OSD3_DEVICE}
      OSD_TYPE: disk
      OSD_FORCE_ZAP: 1
    depends_on:
      - mon1
    networks:
      - ceph-cluster-net

  rgw1:
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "rgw"
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
    depends_on:
      - osd1
      - osd2
      - osd3
    networks:
      - ceph-cluster-net
    ports:
      - ${RGW_PORT}:8080

  mds1:
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "mds"
    environment:
      CEPHFS_CREATE: 1
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
    depends_on:
      - osd1
      - osd2
      - osd3
    networks:
      - ceph-cluster-net

networks:
  ceph-cluster-net:
    driver: bridge
    external: true

I have an env-example file:

CEPH_CONTAINER_VERSION=v3.1.0-stable-3.1-luminous-ubuntu-16.04-x86_64
VOLUMES_PATH=~/volumes_data/ceph
DASHBOARD_PORT=6000
RGW_PORT=6080
OSD1_DEVICE=/dev/sdb
OSD2_DEVICE=/dev/sdc
OSD3_DEVICE=/dev/sdd
MON1_IP=172.18.0.2
MON1_CEPH_PUBLIC_NETWORK=172.18.0.0/24

Before running docker-compose, I run these commands:

docker network create --driver bridge ceph-cluster-net
cp env-example .env
docker-compose up -d

Logs for not running images look like this: MON1:

importing contents of /var/lib/ceph/bootstrap-osd/ceph.keyring into /etc/ceph/ceph.mon.keyring
importing contents of /var/lib/ceph/bootstrap-mds/ceph.keyring into /etc/ceph/ceph.mon.keyring
importing contents of /var/lib/ceph/bootstrap-rgw/ceph.keyring into /etc/ceph/ceph.mon.keyring
importing contents of /var/lib/ceph/bootstrap-rbd/ceph.keyring into /etc/ceph/ceph.mon.keyring
importing contents of /etc/ceph/ceph.client.admin.keyring into /etc/ceph/ceph.mon.keyring
2019-12-24 12:18:53.293858 7ff156f2a080 -1 unable to find any IP address in networks '172.18.0.0/24' interfaces ''

MGR:

2019-12-24 12:18:54  /entrypoint.sh: static: does not generate config
2019-12-24 12:18:54  /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2019-12-24 12:18:54  /entrypoint.sh: static: make it available with the help of your configuration management system.
2019-12-24 12:18:54  /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2019-12-24 12:18:54  /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2019-12-24 12:18:54  /entrypoint.sh: static: if you're interested, please visit: https://github.com/ceph/ceph-ansible
2019-12-24 12:23:54.568294 7f3373429700  0 monclient(hunting): authenticate timed out after 300
2019-12-24 12:23:54.568338 7f3373429700  0 librados: client.admin authentication error (110) Connection timed out
[errno 110] error connecting to the cluster

OSD1:

2019-12-24 12:18:50  /entrypoint.sh: static: does not generate config
2019-12-24 12:19:00  /entrypoint.sh: Timed out while trying to reach out to the Ceph Monitor(s).
2019-12-24 12:19:00  /entrypoint.sh: Make sure your Ceph monitors are up and running in quorum.
2019-12-24 12:19:00  /entrypoint.sh: Also verify the validity of client.bootstrap-osd keyring.

OSD2 and OSD3:

2019-12-24 12:10:44  /entrypoint.sh: static: does not generate config
2019-12-24 12:10:44  /entrypoint.sh: ERROR- The device pointed by OSD_DEVICE (/dev/sdc) doesn't exist !
2019-12-24 12:18:48  /entrypoint.sh: static: does not generate config
2019-12-24 12:18:48  /entrypoint.sh: ERROR- The device pointed by OSD_DEVICE (/dev/sdc) doesn't exist !

RGW1:

2019-12-24 12:18:52  /entrypoint.sh: static: does not generate config
2019-12-24 12:19:02  /entrypoint.sh: Timed out while trying to reach out to the Ceph Monitor(s).
2019-12-24 12:19:02  /entrypoint.sh: Make sure your Ceph monitors are up and running in quorum.
2019-12-24 12:19:02  /entrypoint.sh: Also verify the validity of client.bootstrap-rgw keyring.

MDS1:

2019-12-24 12:18:53  /entrypoint.sh: static: does not generate config

When creating docker-compose, I used this article: https://github.com/VasiliyLiao/ceph-docker-compose

Answer 1

you dont have to use docker compose , there is ceph-ansible project where you can run the ceph services and OSD services in docker container, its simple to deploy and maintain after it.

https://github.com/ceph/ceph-ansible

Answer 2

I have a same problem. Working around and find out that "mon1" config should have ipv4_address and make sure MON_IP is equal to that ipv4_address. Example:

environment:
  MON_IP: 172.28.0.10
  CEPH_PUBLIC_NETWORK: 172.28.0.0/24
networks:
  ceph_network :
    ipv4_address: 172.28.0.10

I'm not sure this is the right way to fix this problem, but it works for me.

Answer 3

looking at the logs

unable to find any IP address in networks '172.18.0.0/24' interfaces ''

I'd start with checking the network. Maybe even better, if you're weak with devops try ceph-ansible with only few changes you can have your ceph with docker ready.