CODEWITHSUNDEEP
scaladataframeapache-spark
techie
techie

Reputation: 363

Spark join 2 dataframe based on multiple columns

I have 2 dataframes df1 and df2. I have join condition for these dataframes defined in a map. However the join column names are different.

I know that I can do like this, val df3 = df2.join(df1, df2("col1") <=> df1("col5") && df2("col2") <=> df1("col6") && df2("col3") <=> df1("col7") && df2("col4") <=> df1("col8"), "left" )

IS there a way to dynamically create the join condition using values in map.

Upvotes: 1

Views: 1488

Answers (1)

Chema
Chema

Reputation: 2828

You could use 

val df3 = df2.join(df1, Seq("col5", "col6", "col7", "col8"), "left")

if you previously rename columns from df2 because columns must exist on both sides.

Upvotes: 2

Related Questions