CODEWITHSUNDEEP
apache-sparkapache-spark-sql
Upul Bandara
Upul Bandara

Reputation: 5958

Spark 1.4: Spark SQL ANY and ALL functions

I'm working on a project which uses Spark SQL 1.4. It looks like that version is not supporting ANY and ALL functions.

So is possible to write a UDFs for these functions or do we have other workarounds.

Thanks,

Upvotes: 2

Views: 2506

Answers (1)

zero323
zero323

Reputation: 330163

Something like this should do the trick:

import scala.reflect.ClassTag
import org.apache.spark.sql.Column

type BinColOp = (Column, Column) => Column

def check[T](f: BinColOp)(
    col: Column, pred: (Column, T) => Column, xs: Seq[T]) = { 
  xs.map(other => pred(col, other)).reduce(f)
}

val all = check[Column] (_ && _) _
val any = check[Column] (_ || _) _

Example usage:

val df = sc.parallelize(
  (1L, "foo", 3.6) ::
  (2L, "bar", -1.0) ::
  Nil
).toDF("v", "x", "y")

df.select(all($"v", _ > _, Seq(lit(-1), lit(1)))).show
// +---------------------+
// |((v > -1) && (v > 1))|
// +---------------------+
// |                false|
// |                 true|
// +---------------------+

df.select(any($"x", _ !== _, Seq(lit("foo"), lit("baz")))).show
// +--------------------------------+
// |(NOT (x = foo) || NOT (x = baz))|
// +--------------------------------+
// |                            true|
// |                            true|
// +--------------------------------+

df.select(all($"x", _ !== _, Seq(lit("foo"), lit("baz")))).show
// +--------------------------------+
// |(NOT (x = foo) && NOT (x = baz))|
// +--------------------------------+
// |                           false|
// |                            true|
// +--------------------------------+

Upvotes: 2

Related Questions