CODEWITHSUNDEEP
scalaapache-spark
vaer-k
vaer-k

Reputation: 11773

Why does Spark dataset foreach method require explicit function conversion, but map does not?

Where UserData.retrieve is of type (m: Map[String, String]) => User and UserData.update is of type (r: spark.sql.Row) => Unit

why does

receipts map UserData.retrieve

work fine, but when I run

events foreach UserData.update

it results in the following error:

missing argument list for method update in object UserData Unapplied methods are only converted to functions when a function type is expected. You can make this conversion explicit by writing `update _` or `update(_)` instead of `update`.

I see in the documentation that map expects func: (T) ⇒ U and foreach expects f: (T) ⇒ Unit so it seems to me that the compiler should convert my methods to functions for both map and foreach, but instead it only does so for map. Why?

n.b. I know that I can make this foreach call work by explicitly converting it,

receipts.foreach.(UserData.update(_))

but here I am asking for help understanding why I have to.

Upvotes: 1

Views: 83

Answers (1)

Terry Dactyl
Terry Dactyl

Reputation: 1868

Unapplied methods are only converted to functions when a function type is expected.

foreach on Dataset is overloaded and may be called with ForeachFunction[T] or T => Unit.

The compiler does not have enough information to resolve which overloaded method is being called and therefore will not apply the function.

Upvotes: 1

Related Questions