Insert in batch mode on PostgreSQL 9.6 with Apache Spark 1.6

Question

Please, there is a configuration or params to do this:

dataFrame.write.mode(saveMode).jdbc(url, "tablename", new java.util.Properties())

Command is written in Scala, Apache Spark 1.6

That takes a DataFrame and inserts data in PostgreSQL row by row, one by one. We need to do this in batch mode.

Anyone knows if exists any configuration or another way to accomplish that?

Thanks in advance

Answer 1

According to documentation - link - you can use batchsize parameter to perform inserts in batches

dataFrame
    .write
    .mode(saveMode)
    .option("batchsize", 1000)
    .jdbc(url, "tablename", new java.util.Properties())

Now insert will be perfomed in batches of 1000 rows