CODEWITHSUNDEEP
apache-spark
vatsal mevada
vatsal mevada

Reputation: 5636

Create a single CSV string out of a spark dataframe column containing csv values

I have df like below:

df.show()

1,2
1,4 
1,4,5

The data frame have only 1 column and 3 records each having csv string and I want to create following csv string out of above dataframe:

1,2,4,5

Upvotes: 0

Views: 181

Answers (1)

research800
research800

Reputation: 81

Looking at the output, you should collect it on the driver. To do that, use flatMap & toLocalIterator to retrieve elements as a list.

Here is an example:

def csv2rows(x):  
    for ele in x.split(','):  
      yield ele  
df.flatMap(csv2rows).distinct().toLocalIterator()

Upvotes: 3

Related Questions