CODEWITHSUNDEEP
apache-sparkapache-kafkaapache-spark-sqlspark-structured-streaming
BdEngineer
BdEngineer

Reputation: 3199

Can we channel different kind of message through same kafka topic?

I have scenario where I have different type of messages to be streamed from kafka producer.

If I dont want to use different topic per different message type how to handle it at spark-structured-streaming consumer side ?

i.e. only one topic I want to use for different type of messages ...say Student record , Customer record....etc.
How to identify which message is been received from Kafka topic?

Please let me know how to handle this scenario at kafka consumer side?

Upvotes: 3

Views: 1921

Answers (2)

Ged
Ged

Reputation: 18098

Yes you can do this by adding "some attribute" to the message itself when producing which signifies a logical topic, or operation, and differentiating on the Spark side - e.g. Structured Streaming KAFKA integration. E.g. checking the message content for "some attribute" and process accordingly.

Partitioning is used of course for ordering always.

Upvotes: 2

OneCricketeer
OneCricketeer

Reputation: 191884

Kafka topics don't inheriently have "types of data". It's all bytes, so yes you can serialize completely separate objects into the same topic, but consumers must then add logic to know what are all possible types will get added into the topic.

That being said, Structured Streaming is built on the idea of having structured data with a schema, so it likely will not work if you had completely different types in the same topic without at least performing a filter first based on some inner attribute that is always present among all types.

Upvotes: 3

Related Questions