CODEWITHSUNDEEP
pythonapache-sparkdataframerow
Vinicius Morais
Vinicius Morais

Reputation: 31

Union Row inside Row PySpark Dataframe

I want to convert my Dataframe that has rows inside rows to a unique row, like this:

My dataframe:

[Row(Autorzc=u'S', Cd=u'00000012793', ClassCli=u'A' Op=Row(CEP=u'04661904', CaracEspecial='S', Venc=Row(v110=u'1', v120=u'2'))),
Row(Autorzc=u'S', Cd=u'00000012794', ClassCli=u'A' Op=Row(CEP=u'04661904', CaracEspecial='S', Venc=Row(v110=u'1', v120=u'2')))]

and I want to transform to this:

[Row(Autorzc=u'S', Cd=u'00000012793', ClassCli=u'A', CEP=u'04661904', CaracEspecial='S', v110=u'1', v120=u'2'),
Row(Autorzc=u'S', Cd=u'00000012794', ClassCli=u'A', CEP=u'04661904', CaracEspecial='S', v110=u'1', v120=u'2')]

Any suggestion?

Upvotes: 1

Views: 152

Answers (1)

user238607
user238607

Reputation: 2468

You can do a simple select operation and your columns will be renamed accordingly.

final = initial.select("Autorzc","Cd" , "ClassCli", "Op.CEP"
       "Op.CaracEspecial","Op.Venc.v110","Op.Venc.v120")

print(final.first())

Upvotes: 1

Related Questions