CODEWITHSUNDEEP
pythonapache-sparkpyspark
rosefun
rosefun

Reputation: 1867

Pyspark: convert tuple type RDD to DataFrame

I hava a complicated tuple type RDD, like

(20190701, [11,21,31], [('A',10), ('B', 20)])

The schema can be defined by myself.

So how to turn it into a DataFrame, like this:

date | 0 | 1 | 2 | A | B 
20190701 | 11 | 21 | 31 | 10 | 20

Upvotes: 0

Views: 276

Answers (2)

Digital.Code
Digital.Code

Reputation: 1

rdd = sc.parallelize((20190701, [11,21,31], [('A',10), ('B', 20)]))

elements = rdd.take(3)

a = [elements[0]] + (elements[1]) + [elements[2][0][1], elements[2][1][1]]

import pandas as pd
sdf = spark.createDataFrame(pd.DataFrame([20190701, 11, 21, 31, 10, 20]).T)

Upvotes: 0

jxc
jxc

Reputation: 14008

One way:

from pyspark.sql import Row

rdd = sc.parallelize([(20190701, [11,21,31], [('A',10), ('B', 20)])])

# customize a Row class based on schema    
MRow = Row("date", "0", "1", "2", "A", "B")

rdd.map(lambda x: MRow(x[0], *x[1], *map(lambda e:e[1],x[2]))).toDF().show()
+--------+---+---+---+---+---+
|    date|  0|  1|  2|  A|  B|
+--------+---+---+---+---+---+
|20190701| 11| 21| 31| 10| 20|
+--------+---+---+---+---+---+

Or another way:

rdd.map(lambda x: Row(date=x[0], **dict((str(i), e) for i,e in list(enumerate(x[1])) + x[2]))).toDF().show()
+---+---+---+---+---+--------+
|  0|  1|  2|  A|  B|    date|
+---+---+---+---+---+--------+
| 11| 21| 31| 10| 20|20190701|
+---+---+---+---+---+--------+

Upvotes: 2

Related Questions