CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Selnay
Selnay

Reputation: 711

Different rlike behavior in Spark 1.6 and Spark 2.2

I am applying some filters to a dataframe in Spark 1.6 and Spark 2.2 and I am getting a totally different behavior. The code I am executing is:

val df = Seq(
  (1, 2),
  (3, 4),
  (5, 6),
  (7, 8)
).toDF("col1", "col2")

val filter = "col1 rlike '[\\x00-\\x1F\\x7F]'"
df.filter(filter).count
// This gives me 0 in Spark 1.6 and 4 in Spark 2.2

val filter2 = "col1 rlike '[\\\\x00-\\\\x1F\\\\x7F]'"
df.filter(filter2).count
// This gives me 4 in Spark 1.6 and 0 in Spark 2.2

Just in case it is important, Spark 1.6 is being executed along with Scala 2.10.5 and Spark 2.2 with 2.11.8.

Any idea of why this may be happening?

Upvotes: 0

Views: 91

Answers (1)

ollik1
ollik1

Reputation: 4540

The behaviour changed with this bug fix https://issues.apache.org/jira/browse/SPARK-17647

Upvotes: 3

Related Questions