Spring Cloud Stream (Hoxton) Kafka Producer / Consumer not working in integration tests with EmbeddedKafka

Question

I have a working application that uses the latest update for Producers that came with Hoxton. Now I'm trying to add some integration tests, asserting that the Producer is actually producing a message as expected. The problem is, the consumer I use in the test never reads anything from the topic.

In order to make this issue reproducible I've resused a project (spring-cloud-stream-samples/source-samples/dynamic-destination-source-kafka) from the spring cloud stream samples, adapting it as follows:

DynamicDestinationSourceApplication (The EmitterProcessor is now a bean)

@SpringBootApplication
@RestController
public class DynamicDestinationSourceApplication {

    @Autowired
    private ObjectMapper jsonMapper;

    @Autowired
    private EmitterProcessor<Message<?>> processor;

    public static void main(String[] args) {
        SpringApplication.run(DynamicDestinationSourceApplication.class, args);
    }

    @SuppressWarnings("unchecked")
    @RequestMapping(path = "/", method = POST, consumes = "*/*")
    @ResponseStatus(HttpStatus.ACCEPTED)
    public void handleRequest(@RequestBody String body, @RequestHeader(HttpHeaders.CONTENT_TYPE) Object contentType) throws Exception {
        Map<String, String> payload = jsonMapper.readValue(body, Map.class);
        String destinationName = payload.get("id");
        Message<?> message = MessageBuilder.withPayload(payload)
                .setHeader("spring.cloud.stream.sendto.destination", destinationName).build();
        processor.onNext(message);
    }

    @Bean
    public Supplier<Flux<Message<?>>> supplier() {
        return () -> processor;
    }

    @Bean
    public EmitterProcessor<Message<?>> processor(){
        return EmitterProcessor.create();
    }

    //Following sink is used as test consumer. It logs the data received through the consumer.
    static class TestSink {

        private final Log logger = LogFactory.getLog(getClass());

        @Bean
        public Consumer<String> receive1() {
            return data -> logger.info("Data received from customer-1..." + data);
        }

        @Bean
        public Consumer<String> receive2() {
            return data -> logger.info("Data received from customer-2..." + data);
        }
    }
}

ModuleApplicationTests

@EmbeddedKafka
@RunWith(SpringJUnit4ClassRunner.class)
@SpringBootTest(classes = DynamicDestinationSourceApplication.class)
@WebAppConfiguration
@DirtiesContext
public class ModuleApplicationTests {

    private static String TOPIC = "someTopic";

    @Autowired
    private EmbeddedKafkaBroker embeddedKafkaBroker;

    @Autowired
    private EmitterProcessor<Message<?>> processor;

    @Test
    public void shouldProduceAndConsume() {

        Map<String, Object> configs = new HashMap<>(KafkaTestUtils.consumerProps("consumer", "false", embeddedKafkaBroker));
        Consumer<String, String> consumer = new DefaultKafkaConsumerFactory<>(configs, new StringDeserializer(), new StringDeserializer()).createConsumer();
        consumer.subscribe(Collections.singleton(TOPIC));
        consumer.poll(0);

        Message<?> message = MessageBuilder.withPayload(new HashMap<String,String>(){{put("somekey", "somevalue");}})
                .setHeader("spring.cloud.stream.sendto.destination", TOPIC).build();
        processor.onNext(message);

        ConsumerRecord<String, String> someRecord = KafkaTestUtils.getSingleRecord(consumer, TOPIC);
        System.out.println(someRecord);

    }

}

It ends with No records found for topic. Why isn't this working during the tests?

UPDATE:

My actual project doesn't behave exactly as the project above, what I see that emitterProcessor.onNext() doesn't end up invoking AbstractMessageHandler.onNext()

Debugging into emitterProcessor.onNext() I've seen that it calls drain() and in FluxPublish.PubSubInner<T>[] a = subscribers; subscribers is an empty array, whereas in normal application execution it contains an EmitterProcessor.

Answer 1

I had testImplementation("org.springframework.cloud:spring-cloud-stream-test-support") incorrectly added as a dependency. This uses a Test Binder that is not meant to be used with integration tests.