CODEWITHSUNDEEP
jsonscalapysparktransformationazure-databricks
anuj
anuj

Reputation: 134

Transformation and coding in Pyspark or Scala scenario

I have a situation in my json file I have two columns eventid & sectionname which is dynamic in nature. As mentioned in diagram input

enter image description here enter image description here

I need output like this which transformation I can perform and since section name is dynamic i.e instead of 301 it will come 501 also in future & I don't want my stream to fail is there any way in pyspark or scala. enter image description here

Upvotes: 1

Views: 53

Answers (1)

Aman Mishra
Aman Mishra

Reputation: 26

df_target = (df_source.set_index(list of static columns)
  .rename_axis([New_Column_Name], axis=1)
  .stack()
  .reset_index())

Where df_source is your dataframe in pandas

Upvotes: 1

Related Questions