CODEWITHSUNDEEP
dataframepysparktransformationrdd
W4K1NG
W4K1NG

Reputation: 247

Convert PySpark DataFrame back to rows

I have some existing code which relies on data being in a row ala: [u'0,1,1,5,0,1382,4,15]

In order to make some transformations, I had to convert my RDD to a dataframe sp it now looks like this:

Row(a=u'1', code=u'ts=12206384',date=u'2014-10-05', cstat='200', 'substat'=0,', time=0, time=u'00:06:18' Target=0)]

Is it possible to convert the spark DF back to it's original row format so that the rest of my code will work?

Upvotes: 0

Views: 869

Answers (1)

Katya Willard
Katya Willard

Reputation: 2182

I'm going to assume you mean you want to get from a Row object back to a single string of comma separated values.

You would take your dataframe which contains Row objects and do the following:

df_of_row_objects.map(lambda row: ",".join(x for x in row))

This code iterates through each Row in your dataframe and joins each item in the row by a comma.

Upvotes: 1

Related Questions