CODEWITHSUNDEEP
scalaapache-sparkdictionaryjoinreduce
Michael Blankenship
Michael Blankenship

Reputation: 27

How to join a map of DataFrames? Scala Spark

I have a map of DataFrames:

val myMap = Map(
    "name1" -> df1,
    "name2" -> df2,
    "name3" -> df3
)

I want to join these DataFrames on a list of common columns:

val commonColumns = Seq("id", "time")

And so what statement can I write to get the effect of the below for any length of map of df's?:

val desiredDf = df1.join(df2, commonColumns).join(df3, commonColumns)

Upvotes: 0

Views: 1321

Answers (1)

Travis Hegner
Travis Hegner

Reputation: 2495

How about this:

val desiredDf = myMap.values.reduce((l, r) => {
  l.join(r, commonColumns)
})

Upvotes: 1

Related Questions