How to route with traefik to one docker swarm service exposing two ports?

Question

RabbitMq serves a management GUI at port 15672 and clients connect to the message broker at port 5672

My environment:
docker swarm Server Version: 19.03.5
image: "traefik:v2.0.2"
image: rabbitmq:management-alpine

I can browse the Rabbit MQ management pages at https://mq.mydom.comexample With this below as part of my compose.yml for docker stack deploy

  mq:
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.mq-service.rule=Host(`mq.mydom.comexample`)"
        - "traefik.http.services.mq-service.loadbalancer.server.port=15672"
        - "traefik.http.routers.mq-service.entrypoints=websecure"
        - "traefik.http.routers.mq-service.tls.certresolver=mytlschallenge"

Now I also wish to connect to the RabbitMq message broker in the same service But I do not understand the traefik syntax. Tried variations without success. Here is one of those variations which may show what I would have liked to achieve:

  mq:
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.mq-service.rule=Host(`mq.mydom.comexample`)"
        - "traefik.http.services.mq-service.loadbalancer.server.port=15672"
        - "traefik.http.routers.mq-service.entrypoints=websecure"
        - "traefik.http.routers.mq-service.tls.certresolver=mytlschallenge"
        # so far same as above

        - "traefik.http.routers.mq-connect.rule=Host(`mq-connect.mydom.comexample`)"
        - "traefik.http.services.mq-connect.loadbalancer.server.port=5672"

browsing https://mq.mydom.comexample or https://mq-connect.mydom.comexample both respond 404 page not found

telnet mq.mydom.comexample 443
telnet mq-connect.mydom.comexample 443
both connect with: Escape character is '^]'.

How do I tell traefik routing what I want?

Answer 1

I stumbled across this answer while trying to get RabbitMQ to run behind Traefik. After wrestling for days to get these solutions to run on my EC2 instance, I finally realized that the only difference between these examples (which work perfectly) and the way I was running them on the cloud were the docker resource constraints (which I always apply to cloud services).

I don't know what the minimum constrains are that will work (or how these constraints interfere with RabbitMQ badly), but I can say that these resource limits cause issues and would created a semi-functional RMQ instance (I could often connect to it, but could not successfully pull tasks from it). I thought this was because of some Traefik routing I had messed up, but it was the docker resource limits noted here. Hope this saves somebody else some headache!

resources:
  limits:
    cpus: "0.25"
    memory: 128M

It appears from Rabbit's Production Checklist that at least 256MB of memory are required. Upping this limit to 256MB did not solve the issue, but a careful reading of this documentation might uncover the resource limitation that was creating the unstable instance.

And it appears from Rabbit's Docker Hub Page that since Rabbit explicitly tracks and manages memory usage, it needs to be made aware of cgroup-imposed limits. There's the issue! You must let Rabbit know about any docker imposed resource limitations to memory.

Answer 2

To summarise, this now works for me (many thanks).

version: "3.7"

  traefik:
    image: "traefik:v2.0.2"
    command:
      - "--api.insecure=true"
      - "--providers.docker.swarmMode=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.rabbitmq.address=:5672"
      - "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
     # testing: Letsencrypt would close us out for 24hrs if too many bad requests are made!
     #- "--certificatesresolvers.mytlschallenge.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.mytlschallenge.acme.email=postmaster@mydomain"
      - "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
      - "--log=true"
      - "--log.filepath=/var/log/traefik.log"
    ports:
      # https
      - "443:443"
      # traefik dash
      - "8080:8080"
      # rabbit mq message broker
      - "5672:5672"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  mq:
    hostname: xyz_mq
    image: rabbitmq:3.8.2-alpine
    environment:
      - RABBITMQ_DEFAULT_VHOST=prod_vhost
      - RABBITMQ_DEFAULT_USER=prod_user
      - RABBITMQ_DEFAULT_PASS=abcdef123
    deploy:
      labels:
        - "traefik.enable=true"

        # web browser access to RabbitMq management GUI
        - "traefik.http.routers.mq.rule=Host(`mq.mydomain`)"
        - "traefik.http.services.mq.loadbalancer.server.port=15672"
        - "traefik.http.routers.mq.entrypoints=websecure"
        - "traefik.http.routers.mq.tls.certresolver=mytlschallenge"
        - "traefik.http.routers.mq.service=mq"

        # AMQPS - SSL terminated by traefik
        - "traefik.tcp.routers.mq-connect.rule=HostSNI(`mq-connect.mydomain`)"
        - "traefik.tcp.services.mq-connect.loadbalancer.server.port=5672"
        - "traefik.tcp.routers.mq-connect.entrypoints=rabbitmq"
        - "traefik.tcp.routers.mq-connect.tls.certresolver=mytlschallenge"
        - "traefik.tcp.routers.mq-connect.service=mq-connect"
    volumes:
      - rabbitmq:/var/lib/rabbitmq
      - ./rabbitmq_plugins:/etc/rabbitmq/enabled_plugins
    stop_grace_period: 5m

volumes:
  rabbitmq:

Answer 3

I have done some more research regarding Traefik V2. The solution for multiple endpoints is to explicitly name your services. Here is an example:

  whoami:
    image: "containous/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.docker`)"
      - "traefik.http.routers.whoami.entrypoints=web"
      - "traefik.http.routers.whoami.service=whoami"
      - "traefik.http.services.whoami.loadbalancer.server.port=80"
      - "traefik.http.routers.altwhoami.rule=Host(`alt.docker`)"
      - "traefik.http.routers.altwhoami.entrypoints=web"
      - "traefik.http.routers.altwhoami.service=altwhoami"
      - "traefik.http.services.altwhoami.loadbalancer.server.port=80"

So in order to fix your traefik implementation you need to add the traefik.http.routers.whoami.service=service-name labels.

Hope this helps.

Edit:

In order to route AMQP over traefik you need a TCP router. I created a small working example for rabbitmq:

version: "3.3"
services:
  traefik:
    image: "traefik"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.rabbitmq.address=:5672"
    ports:
      - "80:80"
      - "8080:8080"
      - "5672:5672"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  rabbitmq:
    image: "rabbitmq:management-alpine"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.rabbitmq.rule=Host(`rabbitmq.docker`)"
      - "traefik.http.routers.rabbitmq.entrypoints=web"
      - "traefik.http.routers.rabbitmq.service=rabbitmq"
      - "traefik.http.services.rabbitmq.loadbalancer.server.port=15672"

      - "traefik.tcp.routers.ingress.rule=HostSNI(`*`)"
      #- "traefik.tcp.routers.ingress.rule=HostSNI(`ingress.docker`)"
      - "traefik.tcp.routers.ingress.entrypoints=rabbitmq"
      #- "traefik.tcp.routers.ingress.tls=true"
      #- "traefik.tcp.routers.ingress.tls.passthrough=true"
      - "traefik.tcp.services.ingress.loadbalancer.server.port=5672"

This will route TCP traffic on port 5672 over traefik to your container (Make sure you adjust your traefik config accordingly).

You might notice the rather open HostSNI(*) rule. If you want to restrict this rule to a single host/domain you will have to enable TLS support in rabbitmq in order for traefik to filter these requests properly.

Check the commented lines on information about converting traefik to support tls. You can either let traefik handle TLS or pass it through to rabbitmq directly.