CODEWITHSUNDEEP
apache-kafkaspark-streaming
theMadKing
theMadKing

Reputation: 2074

Spark-Stream Differentiating Kafka Topics

If I feed my Spark-Streaming Application more than one topic like so:

val ssc = new StreamingContext(sc, Seconds(2))
    val topics = Set("raw_1", "raw_2)
    val kafkaParams = Map("metadata.broker.list" -> "localhost:9092")
    val stream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics)

When I run my application how can I figure out from my stream what the difference between which topic it is pulling from? Is there a way to do? If I do something like

val lines = stream.print()

I am getting nothing of differentiation. Is the only way to do it to make the Kafka Message Key a denoting factor?

Upvotes: 0

Views: 333

Answers (1)

avr
avr

Reputation: 4883

Yes, you can use MessageAndMetadata version of createDirectStream which allows you to access message metadata.

You can find example implementation here .

Upvotes: 1

Related Questions