CODEWITHSUNDEEP
sqlscalaapache-sparkapache-spark-sqlapache-spark-dataset
HDev007
HDev007

Reputation: 325

Spark count number of words with in group by

I have a dataset 

+----------+--------+------------+
|        id|    date|       errors|
+----------+--------+------------+
|1         |20170319|      error1|
|1         |20170319|      error2|
|1         |20170319|      error2|
|1         |20170319|      error1|
|2         |20170319|        err6|
|1         |20170319|       error2|

Need the number error counts day wise

output

+----------+--------+------------+
|    date|       errors| count
+----------+--------+------------+
 |20170319|      error1|    2
 |20170319|      error2|    3
 |20170319|        err6|    1

    val dataset = spark.read.json(path);
    val c =dataset.groupBy("date").count()

//how I proceed to count errors

I tried Windowing over date in spark scala sql but not able find productive do i need to convert to Rdd and find a approach.?

Upvotes: 3

Views: 717

Answers (1)

Vamsi Prabhala
Vamsi Prabhala

Reputation: 49260

You just need to groupBy both date and errors.

val c =dataset.groupBy("date","errors").count()

Upvotes: 1

Related Questions