CODEWITHSUNDEEP
javaapache-kafkaspring-kafka
Neeraj Jain
Neeraj Jain

Reputation: 7730

Kafka's JsonDeserializer not working for java.util.Map

I am using JsonDeserializer to deserialize my custom Object, but in my method annotated with @KafkaListener get the object with Map field as null.

public ConsumerFactory<String, BizWebKafkaTopicMessage> consumerFactory(String groupId) {
  Map<String, Object> props = new HashMap<>();
  props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
  props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
  return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new JsonDeserializer<>(BizWebKafkaTopicMessage.class));
}

and my BizWebKafkaTopicMessage is 

@Data
public class BizWebKafkaTopicMessage {

    // Elastic Search Index Name
    private String indexName;

    // ElasticSearch Index's type name
    private String indexType;

    // Source document to be used
    private Map<String, Object> source; <=== This is being delivered as null.

    // ElasticSearch document primary id
    private Long id;
}

and the listener method listenToKafkaMessages

@KafkaListener(topics = "${biz-web.kafka.message.topic.name}", groupId = "${biz-web.kafka.message.group.id}")
public void listenToKafkaMessages(BizWebKafkaTopicMessage message) {
............................................
............................................
         // Here message.source is null
............................................
............................................
}

Inside listenToKafkaMessages method, message argument looks like this 

message.indexName = 'neeraj';
message.indexType = 'jain';
message.id = 123;
message.source = null;

Upvotes: 0

Views: 1187

Answers (1)

Felteh
Felteh

Reputation: 53

My strong suspicion would be that it is the polymorphic nature of your value rather than Maps per-se.

Spring is using Jackson underneath the hood for it's serialisation/deserialisation - and by default Jackson (in serialisation) when handling Object instances does not encode what class it is serialising.

Why? Well it makes for bad compatibility issues e.g. you serialised an Object (Really MyPojoV1.class) into the database 1 year ago, and then later read it out - but your code has no MyPojoV1.class anymore because things have moved on... It can even cause issues if you move MyPojoV1 to a different package anywhere within the lifetime of your application!

So when it comes to deserialising Jackson doesn't know what class to deserialise the Object into.

A hacky idea would be to run the following somewhere:

ObjectMapper.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);

Or the nicer/more spring way would be to:

@Configuration
public class JacksonConfiguration {

 @Bean
 public ObjectMapper objectMapper() {
    ObjectMapper mapper = new ObjectMapper();
    #Your Configuration Here for example  mapper.configure(DeserializationFeature.something, true);

    return mapper;
 }
}

Finally it's worth adding that deserialising classes arbitrarily is generally a big security risk. There exists classes in Java which execute command line or even reflection based logic based on the value in their fields (which Jackson will happily fill for you). Hence someone can craft JSON such that you deserialise into a class that basically executes whatever command is in the value={} field.

You can read more about exploits here - although I recognise it may not concern you since your Kafka cluster and it's producers may inherently be within your 'trusted boundaries': https://www.nccgroup.trust/globalassets/our-research/us/whitepapers/2018/jackson_deserialization.pdf

Upvotes: 2

Related Questions