CODEWITHSUNDEEP
scalahashmapapache-spark-sql
experiment
experiment

Reputation: 315

how to pick the rows from dataframe comparing with hashmap

I have two dataframes,

df1

      id         slt     sln       elt      eln        start      end

enter image description here

df2

     id           evt         slt    sln     speed     detector

enter image description here

Hashmap

Map(351608084643945 -> List(1544497916,1544497916), 351608084643944 -> List(1544498103,1544498093))

I want to compare the values in the list and if the two values in the list match ,then I want to have the full row from dataframe(df1) of that id. else,full row from df2 of that id.

Both the dataframes and maps will have distinct and unique id.

Upvotes: 0

Views: 301

Answers (2)

Jiayi Liao
Jiayi Liao

Reputation: 1009

Two steps:

  1. Step One: Split the hashmap into two hashmaps, one is the matched hashmap, the other is not matched hashmap.
  2. Step Two: Use matched hashmap to join with df1 on id, then you get the matched df1. And use unmatched hashmap to join with df2 on id, then you get the unmatched df2.

Upvotes: 0

Rishi Saraf
Rishi Saraf

Reputation: 1812

If I understand correctly you want to traverse your hash map and for entry you want to check if value which is list have all the values same. If list have same element that you want data from df1 else from df2 for that key. If that is what you want than below is the code for same. 

hashMap.foreach(x => {
        var key = x._1.toString
        var valueElements = x._2.toList
        if (valueElements.forall(_ == valueElements.head)) {
          df1.filter($"id".equalTo(key))
        } else {
          df2.filter($"id".equalTo(key))
        }
      })

Upvotes: 1

Related Questions