CODEWITHSUNDEEP
dockerdocker-composedocker-swarmgrpctraefik
Vag Karv
Vag Karv

Reputation: 73

Trying to connect to a grpc service, running on an isolated Docker swarm, that is exposed through Traefik

I have been trying to configure Traefik to be the reverse proxy for my dev docker swarm which runs on its own VPN, isolated from the outer world. The swarm consists of 3 manager nodes and 3 worker nodes and it does not have any TLS for communication between them. I have managed to properly set a number of frontends (with rules for Host) and backends that point to services running in the swarm. Below, is my docker-compose file which I use to deploy the docker stack for traefik on the swarm.

version: '3.6'

networks:
  traefik:
    driver: overlay

configs:
  traefik:
    file: ./config.dev.toml

services:
  traefik:
    image: traefik:1.7
    command:
      - --docker
      - --docker.swarmMode
      - --docker.domain=traefik.dev.pap
      - --docker.exposedByDefault=false
      - --entryPoints=Name:http Address::80 Redirect.EntryPoint:https
      - --entryPoints=Name:https Address::443 TLS:/certs/certfile.cert,/certs/keyfile.key
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    configs:
      - source: traefik
        target: /traefik.toml
    networks:
      - traefik
    ports:
      - target: 80
        published: 80
        mode: host
      - target: 443
        published: 443
        mode: host
      - target: 8080
        published: 8080
        mode: host
      - target: 8081
        published: 8081
        mode: host
      - target: 8082
        published: 8082
        mode: host
    deploy:
      mode: global
      placement:
        constraints:
          - node.role == manager
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure

Toml file for traefik

defaultEntryPoints = ["http", "https"]

logLevel = "WARN"

insecureSkipVerify = true

[entryPoints]
  [entryPoints.http]
    address = ":80"
    [entryPoints.http.auth.basic]
    users = ["traefik:$2y$05$ZF08gDSv0f88vEUVzmwuxu.9seYIAZVzTMXnrETBDhQT30dxu4vOu"]
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
    address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = """-----BEGIN CERTIFICATE-----
                    MIICsjCCAZoCCQDP9WReX5fgQjANBgkqhkiG9w0BAQsFADAbMRkwFwYDVQQDDBAq
                    dHJhZWZpay5kZXYucGFwMB4XDTE5MDUyNDIwMTYzMloXDTIwMDUyMzIwMTYzMlow
                    GzEZMBcGA1UEAwwQKnRyYWVmaWsuZGV2LnBhcDCCASIwDQYJKoZIhvcNAQEBBQAD
                    ggEPADCCAQoCggEBAJ86nJMpyENKQrsDlfrMI8TwdAuHXXLpglMSPVmmTzm0Y7F7
                    CVRAwl8NJdbBJnBmK3+vaG1P3WfXfu/pBfn8GDrvlJd+1XqKdExMYMCDy2Ay1ghk
                    1mOcRkJ2XS2h7FjvGjAmE7/icYZprRziek/4hLCtpTjPwPIx/5tN/ZR+lpMXe+dG
                    24CqvcTftZCS+RZSMuACG0tKlmhOx/xjoUOQMiEA8ysmQE+RrEs7ZSvu1SLYMILa
                    jGrHmH/52kIia+zOJqi8k4pVTkRFB1CNeSURgFOQf1zWzSI49GTh0UGsmlgeMoOX
                    FgrEjsbmMKxCz8ItPiTMjJ1zLd5YyBt+ATeDl0kCAwEAATANBgkqhkiG9w0BAQsF
                    AAOCAQEAYXrBbyJ75TMInYmPjkPD/Vdh3dI62qwHrKJsXN519FC8gbYSJFSdr5Pd
                    y/stM2Xn8KGgnkLSuo62MzGD15X/IYqn4Kt9Eizqd5kpsdNc7l/pTTidcCY3nQ82
                    CUGQrnwyVjZ/8wwbjtTL1TOismK3yA169WXD2yXGz15fR08lLCMqXEKotl82FxfX
                    6mw9EkLC3MAfIxuuWcaCl8/AHAUlrzrQOoVj9OkBcxwYeYH++KKYOGCaiuKMen9x
                    cjyCo05qmBQq4PWOoafDJMrdAcnzjc0tTW3DBTnJY7tR0qA/6iGVsdlHce6rMZtR
                    sb5m6XFgABCcg54yfrHIGgv1Te78aA==
                    -----END CERTIFICATE-----
                    """
      keyFile = """-----BEGIN PRIVATE KEY-----
                    MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQCfOpyTKchDSkK7
                    A5X6zCPE8HQLh11y6YJTEj1Zpk85tGOxewlUQMJfDSXWwSZwZit/r2htT91n137v
                    6QX5/Bg675SXftV6inRMTGDAg8tgMtYIZNZjnEZCdl0toexY7xowJhO/4nGGaa0c
                    4npP+ISwraU4z8DyMf+bTf2UfpaTF3vnRtuAqr3E37WQkvkWUjLgAhtLSpZoTsf8
                    Y6FDkDIhAPMrJkBPkaxLO2Ur7tUi2DCC2oxqx5h/+dpCImvsziaovJOKVU5ERQdQ
                    jXklEYBTkH9c1s0iOPRk4dFBrJpYHjKDlxYKxI7G5jCsQs/CLT4kzIydcy3eWMgb
                    fgE3g5dJAgMBAAECggEBAJb3QyxkGPNznQEizTgUBHXcZKnplbg1nNKWIIS3IbZc
                    SYkkVhwDqkXkeVO7nW5D8EoHTMMXTZv+vONJ09jII1D9hax9HHoqe0k7wR/F1TVH
                    /ZXHxVx79kFdZ8q0Exne2L193Iu5blk6KF+SetUHkwbQHC5ZC16V+PqOXzZDLxAV
                    XOXjv5nOK2bqffgZdjYmW3fiwARJ1y/2s6G2Pn3Uk9z3Hk259ZI6k1zE4mwmf62t
                    FQJfjDQ7Ej7N0lXwmSA/Kb7qTXQ/div+juGlEjKTZIZRJMcLOxaZMo93sU8T/Y6b
                    wWnsRvEZlZ9yy8Y/OWN3UxWjc6nx8DPI6sYnTBUNEmECgYEA0Co6Px6n4CFtmeT6
                    6vj/YNGY1vzrycqixtymaIJk7TmPXeLqL919Sro/jbI0B189tnm3DFLk458feRaA
                    Nw+Jl1vQo9IjutJ1ujNt7IrCtlP6smRpQ5fXQ9Ywk3IPtZiG9W+fxM1uHSYKoAh/
                    sAym38Ao5hPRoRJ0TJxvlkCJtz8CgYEAw9GdFa+CXcx9U8pZE1tV2J9gXjd0mr10
                    OKjpKA9ez8IOB9S96bNOiEwHzzsza+OFoou1DrIK0zoK2+XYZYJ+sWOU1pO4PYlj
                    vbRC5kFb2EvD5h47oIoAWqk3QOGewM+z6eWI9r4LTOgdiZ1r6c73jydGGtTBtcZq
                    oqgVOziHV3cCgYEAgtw495se77A0BU3ES1iyujuo+GXRGG7xe1AUsJhgOB06Okw+
                    6k4zyHljIIGJYaUG16KWZwHpcAqFL4rsmTCET4lhlMqyuWw6pA5WqR/mHdaXP/QN
                    lHds1PFeYiKPK5qyJLLCXmbCNULH5SjSlGqvsPNNAq4s1zQRtmOVZoA10uMCgYBw
                    laPrQ+2AYIHuGvj72BeTZI7TnK6+CQ85RGRJVAz/BmNqvTYzl5QfaRdUXoVCEp5S
                    xmBjApqWt1hm9c9lDZIqC0cX1/al6sgd2fZczFcyFN2dAOTneHCbr47FDvNniAO5
                    Bu+obbuxjALvjELvOr+63qm/43M8P8gvnNQYKHgoIQKBgQCQJrT3z27O/hRVUIGg
                    bwAmzwGExOC39uLHNWmOcVmw36oV592sVw1K9pRvPFVJp956T63lhcGv4t0ki2DE
                    gBN/V1jhkAw33qRDd5bbYmIutf6RO7HeewwFHpbaLw0FULe5UlsziE9AbTHmddqU
                    GTcyajXnqimXvSu6AC9XFA9s4g==
                    -----END PRIVATE KEY-----
                   """
  [entryPoints.api]
    address = ":8081"
  [entryPoints.ping]
    address = ":8082"

[api]
  entryPoint = "api"
  dashboard = true

[ping]
  entryPoint = "ping"

[retry]
  logLevel = "DEBUG"

[traefikLog]
  logLevel = "DEBUG"
  format   = "json"

[accessLog]
  logLevel = "DEBUG"
  format = "json"

[docker]
  endpoint = "unix:///var/run/docker.sock"
  domain = "traefik.dev.pbp"
  swarmmode = true
  watch = true
  exposedByDefault = false

The certificates above were generated on my laptop with the command: openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./backend.key -out ./backend.cert and Common Name:*traefik.dev.pap.

The 2 services I want to expose have the following labels:

      labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik_traefik"
      # Public segment
      - "traefik.public.protocol=h2c"
      - "traefik.public.port=50051"
      - "traefik.public.frontend.rule=Host:serviceA-grpc.traefik.dev.pbp"
      - "traefik.public.frontend.auth.forward.tls.insecureSkipVerify=true"
      - "traefik.public.backend.loadbalancer.method=drr"
      - "traefik.public.frontend.passHostHeader=true"

      labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik_traefik"
      # Public segment
      - "traefik.public.protocol=http"
      - "traefik.public.port=8080"
      - "traefik.public.frontend.rule=Host:serviceB-gateway.traefik.dev.pbp"
      - "traefik.public.frontend.passHostHeader=true"
      - "traefik.public.backend.loadbalancer.method=drr"

The first one is a grpc service I am running and the second one is an HTTP gateway for the grpc service. The 2nd one works as expected. There is nothing weird there except the fact that the certificate is insecure. However, I find it really hard to make the grpc service exposed by Traefik. If I execute grpcurl -plaintext tag-service-grpc.traefik.dev.pbp:443 list I get the result Failed to list services: rpc error: code = Unknown desc = Internal Server Error and the Traefic logs are 

{"BackendAddr":"","BackendName":"Traefik","BackendURL":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"x.x.x.x:64391","ClientHost":"x.x.x.x","ClientPort":"64391","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":500,"DownstreamStatusLine":"500 Internal Server Error","Duration":426232,"FrontendName":"Auth for frontend-public-tag-service-public","OriginContentSize":0,"OriginDuration":222656,"OriginStatus":500,"OriginStatusLine":"500 Internal Server Error","Overhead":203576,"RequestAddr":"serviceA-grpc.traefik.dev.pap:443","RequestContentSize":0,"RequestCount":14,"RequestHost":"serviceA-grpc.traefik.dev.pap","RequestLine":"POST /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo HTTP/2.0","RequestMethod":"POST","RequestPath":"/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","RequestPort":"443","RequestProtocol":"HTTP/2.0","RetryAttempts":0,"StartLocal":"2019-05-24T20:24:24.216358162Z","StartUTC":"2019-05-24T20:24:24.216358162Z","level":"info","msg":"","request_Content-Type":"application/grpc","request_Te":"trailers","request_User-Agent":"grpc-go/1.19.0","time":"2019-05-24T20:24:24Z"}
{"BackendAddr":"","BackendName":"Traefik","BackendURL":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"x.x.x.x:64391","ClientHost":"x.x.x.x","ClientPort":"64391","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":500,"DownstreamStatusLine":"500 Internal Server Error","Duration":398352,"FrontendName":"Auth for frontend-public-tag-service-public","OriginContentSize":0,"OriginDuration":142857,"OriginStatus":500,"OriginStatusLine":"500 Internal Server Error","Overhead":255495,"RequestAddr":"serviceA-grpc.traefik.dev.pap:443","RequestContentSize":0,"RequestCount":15,"RequestHost":"serviceA-grpc.traefik.dev.pap","RequestLine":"POST /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo HTTP/2.0","RequestMethod":"POST","RequestPath":"/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","RequestPort":"443","RequestProtocol":"HTTP/2.0","RetryAttempts":0,"StartLocal":"2019-05-24T20:24:24.297632103Z","StartUTC":"2019-05-24T20:24:24.297632103Z","level":"info","msg":"","request_Content-Type":"application/grpc","request_Te":"trailers","request_User-Agent":"grpc-go/1.19.0","time":"2019-05-24T20:24:24Z"}

If I execute grpcurl -plaintext docker-swarm-worker-01.dev.pap:50061 list I get the list of services I can call. Can someone help me please?

Upvotes: 0

Views: 1258

Answers (1)

Vag Karv
Vag Karv

Reputation: 73

I managed to make it work! It seems that the problem was the redirecting of http to https and the basic http authentication that I was using.

Traefik.toml config file now is simpler:

defaultEntryPoints = ["http", "https"]

[entryPoints]
  [entryPoints.http]
    address = ":80"
    compress = true
  [entryPoints.https]
    address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/certs/certfile.cert"
      keyFile = "/certs/keyfile.key"

  [entryPoints.api]
    address = ":8081"
  [entryPoints.ping]
    address = ":8082"

[api]
  entryPoint = "api"
  dashboard = true

[ping]
  entryPoint = "ping"

[retry]

[traefikLog]
  format   = "json"

[accessLog]
  format = "json"

[docker]
  endpoint = "unix:///var/run/docker.sock"
  domain = "traefik.dev.pbp"
  swarmmode = true
  watch = true
  exposedByDefault = false
  insecureSkipVerify = true

Now I can connect to my grpc service running on Docker swarm through Traefik!

Upvotes: 1

Related Questions