Docker and Consul: Pinging services works but curl always returns couldn't connect to host error

Question

I'm trying to deploy multiple services using Docker and use Consul for service discovery. I've been through various guides and tutorials and I have the following bash script that sets up my Docker containers:

export PRIVATE_IP=XXX.XXX.XXX.XXX
export BRIDGE_IP=XXX.XXX.XXX.XXX
docker run -d --name consul -h $HOSTNAME -p $PRIVATE_IP:8300:8300 -p $PRIVATE_IP:8301:8301 -p $PRIVATE_IP:8301:8301/udp -p $PRIVATE_IP:8302:8302 -p $PRIVATE_IP:8302:8302/udp -p $PRIVATE_IP:8400:8400 -p $PRIVATE_IP:8500:8500 -p $BRIDGE_IP:53:53/udp progrium/consul -server -advertise $PRIVATE_IP -bootstrap
docker run -d --name registrator -v /var/run/docker.sock:/tmp/docker.sock -h $HOSTNAME gliderlabs/registrator consul://$PRIVATE_IP:8500
docker run -d --dns=$BRIDGE_IP --dns 8.8.8.8 --dns-search service.consul -p 3579:3579 --name service1 rcarrington/service1
docker run -d --dns=$BRIDGE_IP --dns 8.8.8.8 --dns-search service.consul -p 3580:3580 --name service2 rcarrington/service2

This works in so far as I can navigate to the Consul UI and see all the nodes exist and I can also navigate to the services exposed by service1 and service2. One thing I'm not sure about is that I cannot set the PRIVATE_IP and BRIDGE_IP to anything other than the port the docker server exposes. The script just errors if for example I try to set the BRIDGE_IP to 172.17.42.1 which is apparently the default.

The problem is that I want my services to be able to communicate with each other and they don't seem to be able to do so. If I ping service1 from service2 then I get a response like the following:

# ping service1
# PING service1.service.consul (YYY.YYY.YYY.YYY): ZZ data bytes

This seems to be correct however when I try to use curl to navigate to it's service I get a couldn't connect to host error. I've tried every combination of host names and IP addresses (including the correct port) with no luck. Possibly oddly the IP address that ping returns is not the 1 used in the script above.

I'm running the latest Docker-Toolbox on Windows 10. My service containers are running mono and executing NancyFX self host applications.

I'm assuming I'm doing something wrong in the bash script but I've got no idea what. Any help would be greatly appreciated.

Answer 1

You are using consul in the bridge network and this is why you are facing the issue. Consul will assign a different set of IPs as follows which you will not be able to ping if the other service is out of it.

"TaggedAddresses": {
        "lan": "172.17.0.2",
        "wan": "172.17.0.2"
    },

You have to understand the fact that service discovery is needed when your application is scaling to multiple containers/servers, if you want to run it only on one host for testing you actually can get the benifit of the docker-compose to set all containers in one network bridge and the service discovery will work by default.

So you need to use the --net=host to run your consul containers using the host's network and you will see that the TaggedAddresses are now the host's address.

curl http://localhost:8500/v1/catalog/nodes\?pretty

[
    {
        "ID": "5989c38f-298e-07f3-0069-d448dcc307d7",
        "Node": "consul-master",
        "Address": "10.11.14.171",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "10.11.14.171",
            "wan": "10.11.14.171"
        },
        "Meta": {},
        "CreateIndex": 5,
        "ModifyIndex": 6
    },

You can also dig to veryfy it is working at the DNS level too.

dig @127.0.0.1 -p 8600 stackoverflow.service.dc1.consul. ANY

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.62.rc1.55.amzn1 <<>> @127.0.0.1 -p 8600 stackoverflow.service.dc1.consul. ANY
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65131
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;stackoverflow.service.dc1.consul.  IN  ANY

;; ANSWER SECTION:
stackoverflow.service.dc1.consul. 0 IN  A   10.11.15.112
stackoverflow.service.dc1.consul. 0 IN  A   10.11.14.34
stackoverflow.service.dc1.consul. 0 IN  A   10.11.14.101

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Thu Jul 13 10:13:35 2017
;; MSG SIZE  rcvd: 90

However, you will face the issue with curl here as the dns forwarding does not work on the host network mode with --dns command. Which is not really an issue, as you can use something like dnsmasq or there are already packages that applications can use to look up your service. For example: consul-npm that you can use in your nodejs application

Answer 2

Try using Docker Compose. It's easy to use and has built-in network so containers could communicate with each other.

A simple docker-compose.yml file for your use case looks like (untested):

version: '2'
services:
  consul:
    image: progrium/consul
    container_name: consul
    ports:
      - "8300:8300"
      - "8301:8301"
      - "8302:8302"
      - "8400:8400"
      - "8500:8500"
  registrator:
    image: gliderlabs/registrator
    container_name: registrator
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock
  service1:
    image: rcarrington/service1
    container_name: service1
    ports:
      - "3579:3579"
  service2:
    image: rcarrington/service2
    container_name: service2
    ports:
      - "3580:3580"

All containers are added to network docker_default by default and are able to ping each other via container_name e. g. ping service1.

If you would like to create custom networks you could do:

services:
  consul:
    ...
    networks:
      - consulnet
  registrator:
    ...
    networks:
      - consulnet
networks:
  consulnet

See this link for more information about networking in Docker Compose.