CODEWITHSUNDEEP
pyspark
mommomonthewind
mommomonthewind

Reputation: 4650

pyspark: counting number of occurrences of each distinct values

I think the question is related to: Spark DataFrame: count distinct values of every column

So basically I have a spark dataframe, with column A has values of 1,1,2,2,1

So I want to count how many times each distinct value (in this case, 1 and 2) appears in the column A, and print something like

distinct_values | number_of_apperance
1 | 3
2 | 2

Upvotes: 1

Views: 22054

Answers (2)

cronoik
cronoik

Reputation: 19550

I just post this as I think the other answer with the alias could be confusing. What you need are the groupby and the count methods:

from pyspark.sql.types import *
l = [
1
,1
,2
,2
,1
]

df = spark.createDataFrame(l, IntegerType())
df.groupBy('value').count().show()

+-----+-----+ 
|value|count| 
+-----+-----+ 
|    1|    3|
|    2|    2| 
+-----+-----+

Upvotes: 6

vikrant rana
vikrant rana

Reputation: 4659

I am not sure if you are looking for below solution: Here are my thoughts on this. Suppose you have a dataframe like this.

>>> listA = [(1,'AAA','USA'),(2,'XXX','CHN'),(3,'KKK','USA'),(4,'PPP','USA'),(5,'EEE','USA'),(5,'HHH','THA')]
>>> df = spark.createDataFrame(listA, ['id', 'name','country'])

>>> df.show();
+---+----+-------+
| id|name|country|
+---+----+-------+
|  1| AAA|    USA|
|  2| XXX|    CHN|
|  3| KKK|    USA|
|  4| PPP|    USA|
|  5| EEE|    USA|
|  5| HHH|    THA|
+---+----+-------+

I want to know the distinct country code appears in this particular dataframe and should be printed as alias name.

import pyspark.sql.functions as func
df.groupBy('country').count().select(func.col("country").alias("distinct_country"),func.col("count").alias("country_count")).show()

+----------------+-------------+
|distinct_country|country_count|
+----------------+-------------+
|             THA|            1|
|             USA|            4|
|             CHN|            1|
+----------------+-------------+

were you looking something similar to this?

Upvotes: 3

Related Questions