CODEWITHSUNDEEP
apache-sparkspark-streamingapache-spark-sqlvert.x
Philip K. Adetiloye
Philip K. Adetiloye

Reputation: 3270

How to run multiple spark jobs in parallel with same spark context?

Is there a way to run multiple spark jobs in parallel using the same spark context in different threads ?

I tried using Vertx 3 but looks like each job is being queued up and launch sequentially.

How can I make it run simultaneously with same spark context ?

Here is my sample code:

 vertx.executeBlocking(future -> {
        DataFrame dataframe = sqlContext.sql(sql);

        Row[] result = dataframe.collect();
        System.out.println("Query result for " + sql);
        LOG.info("Query result for " + sql);

        if (result == null) {               
            LOG.info("No result!");
        } else {
            for (Row row : result) {                    
                LOG.info(":::" + row.toString());
            }
        }           
        future.complete(true);
    }, res -> {
        if (res.succeeded()){
            LOG.info("Query finished");
        }else{
            LOG.info("Query failed " + res.cause().getMessage());
            res.cause().printStackTrace();              
        }
    });

Upvotes: 3

Views: 3811

Answers (1)

Justin Pihony
Justin Pihony

Reputation: 67125

How about using the AsyncRDDActions? I just tested and running two collectAsync are indeed run in parallel.

Upvotes: 1

Related Questions