CODEWITHSUNDEEP
apache-sparkcassandraspark-streaming
mananana
mananana

Reputation: 403

Execute multiple actions parallel/async in Spark Streaming

is there a way to execute multiple actions async/parallel in spark streaming? Here is my code:

positions.foreachRDD(rdd -> {

           JavaRDD<A> pbv = rdd.map(p -> A.create(p));          
           javaFunctions(pbv).writerBuilder("poc", "table_a", mapToRow(A.class)).saveToCassandra();     

           JavaRDD<D> pbd = rdd.map(p -> D.create(p));          
           javaFunctions(pbd).writerBuilder("poc", "table_d", mapToRow(D.class)).saveToCassandra();             

           JavaRDD<L> pblv = rdd.map(p -> L.create(p));         
           javaFunctions(pblv).writerBuilder("poc", "table_l", mapToRow(L.class)).saveToCassandra();            

           JavaRDD<V> pbld = rdd.map(p -> V.create(p));         
           javaFunctions(pbld).writerBuilder("poc", "table_v", mapToRow(V.class)).saveToCassandra();

    });

I would like to do the saveToCassandra actions in parallel, is this possible with "spark technics" or only via selfmade Thread/Executer handling?

Thanks for your help!

Regards, Markus

Upvotes: 2

Views: 1172

Answers (1)

RussS
RussS

Reputation: 16576

@massg's comment is probably the fastest approach to this kind of forking but if you don't want that you could more easily do this by just splitting the stream.

Something like

positions.map(A.create(p)).saveToCassandra
positions.map(D.create(p)).saveToCassandra
positions.map(L.create(p)).saveToCassandra
positions.map(V.create(p)).saveToCassandra

By acting on the DSTream all these requests will happen in parallel.

Upvotes: 1

Related Questions