CODEWITHSUNDEEP
apache-sparkdictionarydataframepysparkapache-spark-sql
Toren
Toren

Reputation: 6854

Convert Python dictionary to Spark DataFrame

I have a Python dictionary :

dic = {
       (u'aaa',u'bbb',u'ccc'):((0.3, 1.2, 1.3, 1.5), 1.4, 1),
       (u'kkk',u'ggg',u'ccc',u'sss'):((0.6, 1.2, 1.7, 1.5), 1.4, 2)
       }

I'd like to convert this dictionary to Spark DataFrame with columns :

['key', 'val_1', 'val_2', 'val_3', 'val_4', 'val_5', 'val_6']

example row (1) :

key | val_1 |val_2 | val_3 | val_4 | val_5| val_6|

u'aaa',u'bbb',u'ccc' | 0.3 |1.2 |1.3 |1.5 |1.4 |1 |

Thank you in advance

Upvotes: 2

Views: 11447

Answers (1)

zero323
zero323

Reputation: 330383

Extract items, cast key to list and combine everything into a single tuple:

df = sc.parallelize([
    (list(k), ) + 
    v[0] + 
    v[1:] 
    for k, v in  dic.items()
]).toDF(['key', 'val_1', 'val_2', 'val_3', 'val_4', 'val_5', 'val_6'])

df.show()

## +--------------------+-----+-----+-----+-----+-----+-----+
## |                 key|val_1|val_2|val_3|val_4|val_5|val_6|
## +--------------------+-----+-----+-----+-----+-----+-----+
## |     [aaa, bbb, ccc]|  0.3|  1.2|  1.3|  1.5|  1.4|    1|
## |[kkk, ggg, ccc, sss]|  0.6|  1.2|  1.7|  1.5|  1.4|    2|
## +--------------------+-----+-----+-----+-----+-----+-----+

Upvotes: 8

Related Questions