docker-compose inter container communication

Question

I'm currently experimenting with Spring Boot-based microservices and getting to grips with docker, but I'm hitting a snag.

Basically what I'm trying to do is containerize 2 small services: a spring cloud config service and a spring cloud eureka service (discovery service). The eureka service fetches its configuration from the config service.

Both services are separate projects with their own Dockerfiles:

Dockerfile-cloud-config-service:

FROM openjdk:10.0.2-13-jre-sid
ENV APP_FILE cloud-config-service.jar
ENV APP_HOME /usr/apps
EXPOSE 8888
COPY target/$APP_FILE $APP_HOME/
WORKDIR $APP_HOME
ENTRYPOINT ["sh", "-c"]
CMD ["exec java -jar $APP_FILE"]

Dockerfile-discovery-service:

FROM openjdk:10.0.2-13-jre-sid
ENV APP_FILE discovery-service.jar
ENV APP_HOME /usr/apps
EXPOSE 8761
COPY target/$APP_FILE $APP_HOME/
WORKDIR $APP_HOME
ENTRYPOINT ["sh", "-c"]
CMD ["exec java -jar $APP_FILE"]

Using docker-compose I'm trying to tie them together using the following docker-compose.yml:

version: '3.7'
services:
  cloud-config-service:
    container_name: cloud-config-service
    build:
      context: cloud-config-service
      dockerfile: Dockerfile-cloud-config-service
    image: cloud-config-service:latest
    ports:
      - 8888:8888
    networks:
      - emp-network

  discovery-service:
    container_name: discovery-service
    build:
      context: discovery-service
      dockerfile: Dockerfile-discovery-service
    image: discovery-service:latest
    ports:
      - 8761:8761
    networks:
      - emp-network
    links:
      - cloud-config-service

networks:
  emp-network:
    driver: bridge

At first I configured the discovery-service to fetch its configuration from http://localhost:8888, but after some digging I found that localhost in a container refers to the container itself and found in the Docker documentation that services can refer to each other using their names. So I changed discovery-service's properties to fetch its config from http://cloud-config-service:8888. That doesn't work, hence this post.

Both Dockerfiles build and run just fine, except the fact that the discovery-service fails to GET the config-service on http://cloud-config-service:8888.

It does work if I use the host network driver and the http://localhost:8888 endpoint, but this 'feels' hacky and not how it is supposed to be done.

I'm probably missing something trivial, but I'm afraid I can't find what.

EDIT: Small snippet of discovery-service's console log:

discovery-service       | 2018-10-02 13:14:26.798  INFO 1 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at : http://cloud-config-service:8888
cloud-config-service    | 2018-10-02 13:14:26.836  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$8a18e3b3] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
discovery-service       | 2018-10-02 13:14:27.129  INFO 1 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Connect Timeout Exception on Url - http://cloud-config-service:8888. Will be trying the next url if available
discovery-service       | 2018-10-02 13:14:27.129  WARN 1 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://cloud-config-service:8888/discovery-service/default": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)

Answer 1

Firstly, communication between docker containers is a subset of a much bigger problem prevalent in distributed services - You don't know what service (and hence their dependencies) will go down at any moment, and so you should take into account such failures when you build your application.

The problem that you are facing is common, even more so with Docker containers, and I believe inter-container communication is a major piece in Docker that is under frequent development changes.

To address your problem, first, I would like to put forth some points -

1. localhost from within a container will refer to that container itself.
2. localhost on your machine does in fact refer to your local host and will be mapped with services you map through the ports configuration for each of your services in the docker-compose file.
3. depends_on only waits for the container to start and not for the actual process to start running - which might mean that the service you are waiting for isn't necessarily up and running yet, and so, cause timeouts for the dependent service(s).

What you need is to wait for the service to start running, not just the container to be up. There are two possible ways you can accomplish this -

1. Specify a restart policy for your discovery-service based on failure. In your case, failure would be when it times out while connecting to the cloud-config-service. Something like restart: on-failure:10 which means you are asking docker to restart the discover-service when it fails with a maximum retry of 10. That way, you would have given reasonable time for the other container (service) to be up and running and make sure that the container with the restart policy eventually connects to that one.

2. Use another tool like dockerize that allows you to wait on other services before starting up the container.

Also, to make sure you are debugging the problem correctly, be sure to check the logs of your container to see what the issue really is - docker logs -f --tail=100 <container_name/container_id>.

Hope this helps.

Answer 2

I cannot comment so I'm writing a new answer.

did you try to run something like docker exec -it myconatiner1 curl myconatiner2:port

to see if it sees it or a ping and if not what is the error?

Answer 3

Sounds like you are on right track ... I would introduce notion of depends_on to tell container discovery-service to wait on the booting up of container cloud-config-service before it boots up

  discovery-service:
    container_name: discovery-service
    build:
      context: discovery-service
      dockerfile: Dockerfile-discovery-service
    image: discovery-service:latest
    depends_on:
      - cloud-config-service
    ports:
      - 8761:8761
    networks:
      - emp-network
    links:
      - cloud-config-service

there is a finite period of time required to boot up a container before it can actively handle traffic so its good to attend to this launch sequence, especially true when dealing with say a database container, which others should depends_on