CODEWITHSUNDEEP
scalaapache-spark
touelv
touelv

Reputation: 73

Adding a count column to my sequence in Scala

Given the code below, how would I go about adding a count column? (e.g. .count("*").as("count"))

Final output to look like something like this:

+---+------+------+-----------------------------+------
| id|sum(d)|max(b)|concat_ws(,, collect_list(s))|count|
+---+------+------+-----------------------------+------
|  1|   1.0|  true|                          a. | 1 |
|  2|   4.0|  true|                          b,b| 2 |
|  3|   3.0|  true|                          c. | 1 |

Current code is below:

val df =Seq(
  (1, 1.0, true, "a"),
  (2, 2.0, false, "b")
  (3, 3.0, false, "b")
  (2, 2.0, false, "c")
).toDF("id","d","b","s")

val dataTypes: Map[String, DataType] = df.schema.map(sf => (sf.name,sf.dataType)).toMap

def genericAgg(c:String) = {
  dataTypes(c) match {
    case DoubleType => sum(col(c))
    case StringType => concat_ws(",",collect_list(col(c))) // "append"
    case BooleanType => max(col(c))
  }
}

val aggExprs: Seq[Column] = df.columns.filterNot(_=="id")
.map(c => genericAgg(c))

df
.groupBy("id")
.agg(aggExprs.head,aggExprs.tail:_*)
  .show()

Upvotes: 0

Views: 407

Answers (1)

Leo C
Leo C

Reputation: 22449

You can simply append count("*").as("count") to aggExprs.tail in your agg, as shown below:

df.
  groupBy("id").agg(aggExprs.head, aggExprs.tail :+ count("*").as("count"): _*).
  show
// +---+------+------+-----------------------------+-----+
// | id|sum(d)|max(b)|concat_ws(,, collect_list(s))|count|
// +---+------+------+-----------------------------+-----+
// |  1|   1.0|  true|                            a|    1|
// |  3|   3.0| false|                            b|    1|
// |  2|   4.0| false|                          b,c|    2|
// +---+------+------+-----------------------------+-----+

Upvotes: 1

Related Questions