CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
sparkscala
sparkscala

Reputation: 71

Filter an Array using another Array

   val value = Array["id","sd","cd"]  -- List of columns 
   val cols_list = Array["cd","id","tm","no","in","ts","nm"] -  -- List of columns

i want a list with columns not in cols_list.

code i tried as below :

     val newcol = for (x <- cols_list if x.toString.toUpperCase() not in value )

it's throwing error as value not is not a member of String. is there a way that we can achieve? Please suggest.

Upvotes: 0

Views: 1531

Answers (2)

Ram Ghadiyaram
Ram Ghadiyaram

Reputation: 29195

simplest is filterNot method (Apart from diff) that returns all elements from a list for which your function returns false.

val value = Array("id","sd","cd")  // List of columns
     val cols_list = Array("cd","id","tm","no","in","ts","nm")

     val finallist = cols_list.filterNot(value.contains(_)) //cols_list.par.filterNot also you can use
     println(finallist.mkString(" "))
 }

Result : tm no in ts nm

How it works...

filter creates a collection with those elements that do not satisfy the predicate p and discarding the rest. This is collection level and will work for all Collections API in scala signature : 

  def filterNot(p: (A) => Boolean): Collection[A]

enter image description here

Upvotes: 2

notNull
notNull

Reputation: 31510

use .diff to get list of columns not in cols_list

val value = Array("id","sd","cd")
val cols_list = Array("cd","id","tm","no","in","ts","nm")
value.diff(cols_list)
//Array[String] = Array(sd)

//case insensitive 
value.map(x => x.toUpperCase).diff(cols_list.map(x => x.toUpperCase)).map(x => x.toLowerCase)
//Array[String] = Array(sd)

UPDATE:

cols_list.diff(value)
//Array[String] = Array(tm, no, in, ts, nm)

cols_list.map(x => x.toUpperCase).diff(value.map(x => x.toUpperCase)).map(x => x.toLowerCase)
//Array[String] = Array(tm, no, in, ts, nm)

cols_list.map(x => x.toUpperCase).diff(value.map(x => x.toUpperCase)).map(x => "\"a." + x.toLowerCase + "\"").mkString(",")
//String = "a.tm","a.no","a.in","a.ts","a.nm"

Upvotes: 0

Related Questions