CODEWITHSUNDEEP
pythonapache-spark-sql
2713
2713

Reputation: 195

Pivot a spark dataframe without a groupBy column

Let's say I've a dataframe something like this,

+---+----+------+
|id |name|salary|
+---+----+------+
|10 |abc |100   |
+---+----+------+

And I would like to pivot/transpose the data so that the output looks like,

+--------+----+
|col_name|data|
+--------+----+
|id      |10  |
|name    |abc |
|salary  |100 |
+--------+----+

How would I do this using pyspark.

Upvotes: 2

Views: 205

Answers (2)

Shubham Jain
Shubham Jain

Reputation: 5526

You can use stack as

s = ','.join([f"'{i}', `{i}`" for i in df.columns])
df = df.select([col(i).cast('string') for i in df.columns])
df.select(expr(f'''stack({len(df.columns)},{s})''')).show()

+------+----+
|  col0|col1|
+------+----+
|    id|  10|
|  name| abc|
|salary| 100|
+------+----+

Upvotes: 1

jayrythium
jayrythium

Reputation: 767

I'm not aware of a spark function that does that. You could use expr(stack(...)) or do something similar to this.

Upvotes: 0

Related Questions