CODEWITHSUNDEEP
dataframeapache-sparkpysparkapache-spark-sqldivide
Johanna
Johanna

Reputation: 165

Dividing dataframes in pyspark

Following up this question and dataframes, I am trying to convert this

enter image description here

Into this (I know it looks the same, but refer to the next code line to see the difference):

enter image description here

In pandas, I used the line code teste_2 = (value/value.groupby(level=0).sum()) and in pyspark I tried several solutions; the first one was:

 df_2 = (df/df.groupby(["age"]).sum())

However, I am getting the following error: TypeError: unsupported operand type(s) for /: 'DataFrame' and 'DataFrame'

The second one was:

df_2 = (df.filter(col('Siblings'))/gr.groupby(col('Age')).sum())

But it's still not working. Can anyone help me?

Upvotes: 0

Views: 287

Answers (1)

mck
mck

Reputation: 42352

Hope I've understood the question correctly. It seems you want to divide the count by the sum of count for each age group.

from pyspark.sql import functions as F, Window

df2 = df.groupBy('age', 'siblings').count().withColumn(
    'count',
    F.col('count') / F.sum('count').over(Window.partitionBy('age'))
)

df2.show()
+---+--------+-----+
|age|siblings|count|
+---+--------+-----+
| 15|       0|  1.0|
| 10|       3|  1.0|
| 14|       1|  1.0|
+---+--------+-----+

Upvotes: 1

Related Questions