CODEWITHSUNDEEP
javaspring-bootapache-kafkaspring-kafka
user3248186
user3248186

Reputation: 1558

Consuming from kafka topic using spring only on receiving certain events

I created a kafka consumer in my spring boot application which listens to a topic - my_topic and on reading an event my_event, I perform some action. Right now I'm doing it like this:

@KafkaListener(topics = "my_topic",
          containerFactory = "my_kafka_container_factory")
public void handleMyKafkaEvents(String eventJson) {
    MyDTO my_dto = gson.fromJson(eventJson, MyDTO.class);
    String event_type = my_dto.getEventType();
    if (event_type != null && event_type.equals("my_event")) {
        // do something with my_dto
    }
}

// dto object
public class MyDTO {
    private String status;

    private String eventType;

    private String propName;

    // some other parameters

    // getters and setters
}

A sample of how an object in my kafka topic looks like :

{
  "eventType": "my_event",
  "propName": "prop_value",
  "status": "DONE",
  //some_other_key_value_pairs_required_in_my_DTO
  //some_other_key_value_pairs_not_required_in_my_DTO
}

Since my listener is listening to all data pushed to kafka topic, I had to add a condition after reading each record, that if its eventType is what I need, then I'm performing some action on it.

This's working as of now. Since there'll be other data pushed to it whose eventType isn't what I need, these'll be ignored, but only after reading them, since I didn't know how to filter based on this eventType.

So my question is when there's a sudden spike in the number of events pushed to the kafka topic, not just my eventType, but others as well, will it impact my service's performance?

What can I improve here, so that other eventTypes are ignored and my listener won't even 've to know about them.

Upvotes: 2

Views: 3700

Answers (1)

aran
aran

Reputation: 11900

Some ways that may or not fit into your use case, but may help:

  • Include a mask in the key

Include some specific code into the kafka key, so you don't need to read the payload in order to tell if you must process or not the message.

Just a silly example:

key      payload
-----------------
10_ev  xxx
08_ev  yyy
...

In this simple example, the first two numbers determine the type of event. Each consumer group is assigned a specific event to process, and discards others. Beware! For this to work, you would need to launch as much consumer groups as event types you have, so no message is lost, or assign specific ranges of events to all consumers (f.e, consumer 0 processes event types from 0-9, consumer 1 from 10-19,...)

  • Partition depending on event

You could tell the producer (the one who should know the type of event he's producing) to partition (send the message to specific partition into the topic) the message, so you know, for example, that all events of type 0 are on the partition 0, and consume with that on mind.

Anyway, having too much event types may decrease the options of this to be usable. You could partition based on ranges (event types from 0-9 on partition 0, etc), but well, maybe there's some headache there.

  • Send to different topics depending on event type

Well, this one sure is the most simple one, but may be a issue if you have a lot of event types(like thousands,...). : )

Hope it helps. There's some interesting info about your use case here.

Upvotes: 2

Related Questions