CODEWITHSUNDEEP
apache-sparkjdbcapache-spark-sql
toop
toop

Reputation: 109

Spark - jdbc read all happens on driver?

I have spark reading from Jdbc source (oracle) I specify lowerbound,upperbound,numpartitions,partitioncolumn but looking at web ui all the read is happening on driver not workers,executors. Is that expected?

Upvotes: 0

Views: 1003

Answers (1)

Amit
Amit

Reputation: 1121

In Spark framework, in general whatever code you write within a transformation such as map, flatMap etc. will be executed on the executor. To invoke a transformation you need a RDD which is created using the dataset that you are trying to compute on. To materialize the RDD you need to invoke an action so that transformations are applied to the data.

I believe in your case, you have written a spark application that reads jdbc data. If that is the case it will all be executed on Driver and not executor.

If you haven not already, try creating a Dataframe using this API.

Upvotes: 0

Related Questions