CODEWITHSUNDEEP
javaspringapache-kafkaspring-kafkajson-deserialization
J. Adam
J. Adam

Reputation: 1641

How to deserialize json list with object in my kafka consumer?

My Kafka Producer is sending a list of objects in Json format.

I'm trying to figure out how to make my consumer deserialize the list. I'm able to receive a single object and read it but when i'm change the code to type List i'm getting the following error:

Error:(32, 47) java: incompatible types: cannot infer type arguments for org.springframework.kafka.core.DefaultKafkaConsumerFactory<>
    reason: inference variable V has incompatible equality constraints java.util.List<nl.domain.X>,nl.domain.X

EDIT

This error has been solved by adding TypeReference to the JsonDeserilizer.

Current problem:

When consuming the message, it isn't in the type i defined (which is List< X > ) but returning a LinkedHashMap

This is the Consumer configuration:

@EnableKafka
@Configuration
public class KafkaConfiguration {

    @Bean
    public ConsumerFactory<String, List<X>> xConsumerFactory() {
        Map<String, Object> config = new HashMap<>();

        config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
        config.put(ConsumerConfig.GROUP_ID_CONFIG, "group_json");
        config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
        return new DefaultKafkaConsumerFactory<>(config, new StringDeserializer(),
                new JsonDeserializer<>(new TypeReference<List<X>>() {}));
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, List<X>> xKafkaListenerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, List<X>> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(xConsumerFactory());
        return factory;
    }
}

This is the Consumer:

@Service
public class KafkaConsumer {
    @KafkaListener(topics = "test", groupId = "group_json", containerFactory = "xKafkaListenerFactory")
    public void consume(List<X> x) {
        // In reality it is consuming LinkedHashMap which isn't what i want
        x.forEach(i ->
                System.out.println("Consumed message: " + i.getName()));
    }
}

This is the Producer configuration:

@Configuration
public class KafkaConfiguration {
    @Bean
    public ProducerFactory<String, List<X>> producerFactory() {
        Map<String, Object> config = new HashMap<>();
        config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
        config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);

        return new DefaultKafkaProducerFactory<>(config);
    }

    @Bean
    public KafkaTemplate<String, List<X>> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }
}

This is the producer: 

@Component
public class KafkaProducer {

    private final static String TOPIC = "test";
    private final KafkaTemplate<String, List<X>> kafkaTemplate;

    public KafkaProducer(KafkaTemplate<String, List<X>> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    @Override
    public void publishMessage(List<X> x) {
        // Created 3 instances of X for current example
        List<X> list = new ArrayList();
        list.add(new X("Apple"));
        list.add(new X("Beach"));
        list.add(new X("Money"));
        ListenableFuture<SendResult<String, List<X>>> listenableFuture = kafkaTemplate.send(TOPIC, list);
    }
}

Summary:

The producer seems to work fine. I'm getting the following error in my consumer when sending a message like this. It can't cast LinkedHashMap into List.

org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void nl.infrastructure.input.message.kafka.consumer.KafkaConsumer.consume(java.util.List<nl.domain.X>)' threw exception; nested exception is java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class nl.domain.X (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; nl.domain.X is in unnamed module of loader 'app'); nested exception is java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class nl.domain.X (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; nl.domain.X is in unnamed module of loader 'app')

Upvotes: 1

Views: 13823

Answers (2)

Amber Kulkarni
Amber Kulkarni

Reputation: 443

Using @AmanGarg answer. Tweaking it a bit. (Dont quite know why it did not worked for me.) Converting a single Class to POJO was working but to List<X> was not.
Adding it just if someone faces this issue.

protected JsonDeserializer<List<X>> kafkaDeserializer() {
    ObjectMapper om = new ObjectMapper();
    JavaType type = om.getTypeFactory().constructParametricType(List.class, X.class);
    return new JsonDeserializer<List<X>>(type, om, false);
}

Used different JsonDeserializer constructor.

Upvotes: 1

user9065831
user9065831

Reputation:

It might be because of wrong import type of JsonDeserializer

Use import org.springframework.kafka.support.serializer.JsonDeserializer;

You have to configure JsonDeserializer as below:

protected Deserializer<List<X>> kafkaDeserializer() {
    ObjectMapper om = new ObjectMapper();
    om.getTypeFactory().constructParametricType(List.class, X.class);
    return new JsonDeserializer<>(om);
}

Upvotes: 5

Related Questions