spring MessageConversionException when deserialzing an avro object

Question

I am sending an avro object User and receiving it in a listener. Here are configs

@Bean
public ProducerFactory<String, User> userProducerFactory() {
    Map<String, Object> config = new HashMap<>();

    config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, AvroSerializer.class);

    return new DefaultKafkaProducerFactory<>(config);
}

@Bean
public KafkaTemplate<String, User> userKafkaTemplate() {
    return new KafkaTemplate<>(userProducerFactory());
}

@Bean
public Map<String, Object> consumerConfigs() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, AvroDeserializer.class);
    props.put(ConsumerConfig.GROUP_ID_CONFIG, "avro");

    return props;
}

@Bean
public ConsumerFactory<String, User> consumerFactory() {
    return new DefaultKafkaConsumerFactory<>(consumerConfigs(), new StringDeserializer(),
            new AvroDeserializer<>(User.class));
}

@Bean
public ConcurrentKafkaListenerContainerFactory<String, User> kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, User> factory
            = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());

    return factory;
}

I am sending User like this

@Autowired
KafkaTemplate<String, User> userProducer;
public static void main(String[] args) {
    SpringApplication.run(EvaluatorApplication.class, args);
}

@Override
public void run(String... args) throws Exception {

    User user = User.newBuilder()
                    .setFavoriteColor("red")
                    .setFavoriteNumber(123)
                    .setName("Shantanu")
                    .build();

    userProducer.send("avro.t", user);
}

Here's the Kafka listener

@KafkaListener(topics = "avro.t", containerFactory = "kafkaListenerContainerFactory", groupId = "transaction-request-upstream-consumer")
public void listenUpstream(User user)
{
    try {
        LOGGER.info("Received {}", user);
        messageProcessor.process(null, null);
        //ack.acknowledge();
    } catch (Exception ex) {
        LOGGER.error("{}", ex);
    }
}

In AvroDeserializer class message gets deserialized but then what I get is this exception. I can't make heads or tails of it. Spent a lot of time, totally unable to understand. Please help !!!

[2020-03-26 00:59:16.358] :: [ERROR] :: o.s.k.listener.LoggingErrorHandler.error.254 :: Error while processing: ConsumerRecord(topic = avro.t, partition = 0, leaderEpoch = 0, offset = 4, CreateTime = 1585164556293, serialized key size = -1, serialized value size = 17, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = {"name": "Shantanu", "favorite_number": 123, "favorite_color": "red"})
org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.ibm.dip.evaluator.listeners.EventListener.listenUpstream(com.ibm.dip.evaluator.models.User)]
Bean [com.ibm.dip.evaluator.listeners.EventListener@356dc38d]; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [com.ibm.dip.evaluator.models.User] to [com.ibm.dip.evaluator.models.User] for GenericMessage [payload={"name": "Shantanu", "favorite_number": 123, "favorite_color": "red"}, headers={kafka_offset=4, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@50daf984, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=avro.t, kafka_receivedTimestamp=1585164556293, kafka_groupId=transaction-request-upstream-consumer}], failedMessage=GenericMessage [payload={"name": "Shantanu", "favorite_number": 123, "favorite_color": "red"}, headers={kafka_offset=4, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@50daf984, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=avro.t, kafka_receivedTimestamp=1585164556293, kafka_groupId=transaction-request-upstream-consumer}]; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [com.ibm.dip.evaluator.models.User] to [com.ibm.dip.evaluator.models.User] for GenericMessage [payload={"name": "Shantanu", "favorite_number": 123, "favorite_color": "red"}, headers={kafka_offset=4, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@50daf984, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=avro.t, kafka_receivedTimestamp=1585164556293, kafka_groupId=transaction-request-upstream-consumer}], failedMessage=GenericMessage [payload={"name": "Shantanu", "favorite_number": 123, "favorite_color": "red"}, headers={kafka_offset=4, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@50daf984, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=avro.t, kafka_receivedTimestamp=1585164556293, kafka_groupId=transaction-request-upstream-consumer}]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:1774)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeErrorHandler(KafkaMessageListenerContainer.java:1766)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1679)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:1605)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:1510)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1257)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1007)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:927)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.messaging.converter.MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [com.ibm.dip.evaluator.models.User] to [com.ibm.dip.evaluator.models.User] for GenericMessage [payload={"name": "Shantanu", "favorite_number": 123, "favorite_color": "red"}, headers={kafka_offset=4, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@50daf984, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=avro.t, kafka_receivedTimestamp=1585164556293, kafka_groupId=transaction-request-upstream-consumer}], failedMessage=GenericMessage [payload={"name": "Shantanu", "favorite_number": 123, "favorite_color": "red"}, headers={kafka_offset=4, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@50daf984, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=avro.t, kafka_receivedTimestamp=1585164556293, kafka_groupId=transaction-request-upstream-consumer}]
    at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:336)
    at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:86)
    at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:51)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:1728)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:1711)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1666)
    ... 8 common frames omitted
Caused by: org.springframework.messaging.converter.MessageConversionException: Cannot convert from [com.ibm.dip.evaluator.models.User] to [com.ibm.dip.evaluator.models.User] for GenericMessage [payload={"name": "Shantanu", "favorite_number": 123, "favorite_color": "red"}, headers={kafka_offset=4, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@50daf984, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=avro.t, kafka_receivedTimestamp=1585164556293, kafka_groupId=transaction-request-upstream-consumer}]
    at org.springframework.messaging.handler.annotation.support.PayloadMethodArgumentResolver.resolveArgument(PayloadMethodArgumentResolver.java:145)
    at org.springframework.kafka.annotation.KafkaListenerAnnotationBeanPostProcessor$KafkaNullAwarePayloadArgumentResolver.resolveArgument(KafkaListenerAnnotationBeanPostProcessor.java:906)
    at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:117)
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:148)
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:116)
    at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:48)
    at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:326)
    ... 13 common frames omitted

Answer 1

I think it is caused by dev-tools. I had the same issue like

Caused by: java.lang.ClassCastException: com.ffam.paymentservice.loanadvance.Loanadvance cannot be cast to com.ffam.paymentservice.loanadvance.Loanadvance

And I saw they came from different class loader: one is from Laucher$AppClassLoader, and the other is from RestartClassLoader. After removing dev-tools, everything works fine.

Answer 2

This was a strange one. Out of multiple combinations I tried, I came to know that spring boot version 2.2.5 + spring-boot dev-tools dependency caused this issue. Unable to explain why. But when I switched to spring-boot 2.1.5 and removed dev-tools everything started working fine.

Answer 3

cannot convert from [com.ibm.dip.evaluator.models.User] to [com.ibm.dip.evaluator.models.User]

Generally means it's a class loader issue.

i.e. the User.class in the consumer factory being loaded by a different class loader than the User in the listener bean, so they are effectively different classes.

Try running the JVM with -verbose to see the class loading activity.