CODEWITHSUNDEEP
apache-sparkspark-streamingdstream
Alok
Alok

Reputation: 1506

Spark : get Multiple DStream out of a single DStream

Is is possible to get multiple DStream out of a single DStream in spark. My use case is follows: I am getting Stream of log data from HDFS file. The log line contains an id (id=xyz). I need to process log line differently based on the id. So I was trying to different Dstream for each id from input Dstream. I couldnt find anything related in documentation. Does anyone know how this can be achieved in Spark or point to any link for this.

Thanks

Upvotes: 0

Views: 674

Answers (1)

Sumit
Sumit

Reputation: 1420

You cannot Split multiple DStreams from Single DStreams. The best you can do is: -

  1. Modify your source system to have different streams for different ID's and then you can have different jobs to process different Streams
  2. In case your source cannot change and provide you stream which is mix of ID, then you need to write custom logic to identify the ID and then perform the appropriate operation.

I would always prefer #1 as that is cleaner solution but there are exceptions for which #2 needs to be implemented.

Upvotes: 0

Related Questions