CODEWITHSUNDEEP
mysqlscalacsvapache-sparkjdbc
Ashok Bala
Ashok Bala

Reputation: 168

how to load data from csv to mysql database in Spark?

I would like to load data from csv to mySql as a batch. But I could see the tutorials/logic to insert the data from csv to hive database. Could anyone kindly help me to achieve the above integration in spark using scala?

Upvotes: 0

Views: 4515

Answers (2)

maxmithun
maxmithun

Reputation: 1137

Create the dataframe reading CSV using spark session and write using the method jdbc with mysql Connection properties

val url = "jdbc:mysql://[host][:port][/[database]]"
val table = "mytable"
val property = new Properties()

spark
  .read
    .csv("some/path/to/file.csv")
  .write
    .jdbc(url, table, property)

Upvotes: 1

Sergey Kovalev
Sergey Kovalev

Reputation: 9411

There is a reason why those tutorials don't exist. This task is very straightforward. Here is minimal working example:

val dbStr = "jdbc:mysql://[host1][:port1][,[host2][:port2]]...[/[database]]"

spark
  .read
    .format("csv")
    .option("header", "true")
    .load("some/path/to/file.csv")
  .write
    .mode("overwrite")
    .jdbc(dbStr, tablename, props)

Upvotes: 6

Related Questions