Spring boot container cannot find kafka broker in another container

Question

I have a simple springboot app running in docker container with following configuration.

@Configuration
public class ProducerConfiguration {

    private static final String KAFKA_BROKER = "kafka:9092";

    @Bean
    public ProducerFactory<String, String> producerFactory() {
        return new DefaultKafkaProducerFactory<>(producerConfigurations());
    }

    @Bean
    public Map<String, Object> producerConfigurations() {
        Map<String, Object> configurations = new HashMap<>();

        configurations.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, KAFKA_BROKER);
        configurations.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        configurations.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

        return configurations;
    }

    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }

}

Controller class

@RestController
public class KafkaController {

    private static final Logger LOGGER= LoggerFactory.getLogger(KafkaController.class);

    private KafkaTemplate<String, String> template;

    public KafkaController(KafkaTemplate<String, String> template) {
        this.template = template;
    }

    @GetMapping("/kafka/produce")
    public void produce(@RequestParam String message) {
        template.send("test_topic", message);


    }
}

In the same project, following is the docker-compose.yml for kafka

version: '2.1'

services:
  app:
    restart: always
    build:
      context: ./
      dockerfile: Dockerfile
    image: app
    container_name: app

    ports:
      - 8080
    depends_on:
      - kafka
      - zookeeper
    networks:
      - network1

  kafka:
    image: wurstmeister/kafka
    container_name: 'kafka'
    environment:
      JVM_OPTS: -Xmx1g -Xms1g -XX:MaxPermSize=1024m
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_CREATE_TOPICS: test_topic:1:1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_LISTENERS: INSIDE://kafka:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092

    depends_on:
      - zookeeper
    ports:
      - 9092:9092
      - 9093:9093
    networks:
      - network1


  zookeeper:
    container_name: 'zookeeper'
    image: wurstmeister/zookeeper

    networks:
      - network1

networks:
  network1:

The problem: When I run the springboot application locally and try to send the message to topic, it works fine.but, when I dockerize the springboot application using docker build -t my/app and then docker run --publish=8080:8080 my/app and then try to send the message on topic I get the following error

[Producer clientId=producer-1] Connection to node -1 (kafka/127.0.0.1:9092) could not be established. Broker may not be available.

I have tried various suggestion around KAFKA_ADVERTISED_HOST_NAME but still its not working. Appreciate any help on this.

EDITS : my dockerfile is

FROM maven:3.5.2-jdk-8-alpine AS MAVEN_BUILD
COPY pom.xml /build/
COPY src /build/src/
WORKDIR /build/
RUN mvn package
FROM openjdk:8-jre-alpine
WORKDIR /app
COPY --from=MAVEN_BUILD /build/target/app-0.0.1-SNAPSHOT.jar /app/
ENTRYPOINT ["java", "-jar", "app-0.0.1-SNAPSHOT.jar"]

Answer 1

As per comment, there are a few things to run your spring boot within same network as Kafka.

Find out what network Kafka is under,

docker inspect --format='{{json .NetworkSettings.Networks}}' <kafka_container_id>

response: {"<this_is_network_name>": ...}

To get kafka_container_id a docker ps can be used.

When you know network name, you can do your run command as currently do with:

docker run --network <network_name> --publish=8080:8080  my/app

I've noticed you updated your compose file, the networking should be something like:

networks:
  default:
    external:
      name: my-pre-existing-network

But assigning them to an existing network isn't needed, as they are all contained in same compose, so by default they'll all be assigned the same network, so something like:

version: '2.1'

services:
  app:
    restart: always
    build:
      context: ./
    container_name: app

    ports:
      - 8080:8080
    depends_on:
      - kafka
      - zookeeper

  kafka:
    image: wurstmeister/kafka
    container_name: 'kafka'
    environment:
      JVM_OPTS: -Xmx1g -Xms1g -XX:MaxPermSize=1024m
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_CREATE_TOPICS: test_topic:1:1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_LISTENERS: INSIDE://kafka:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092

    depends_on:
      - zookeeper
    ports:
      - 9092:9092
      - 9093:9093

  zookeeper:
    container_name: 'zookeeper'
    image: wurstmeister/zookeeper

Dockerfile should also expose 8080 (assuming you want to hit the spring boot application)