CODEWITHSUNDEEP
apache-spark
derek
derek

Reputation: 10217

how to use mapPartitions followed by saveAsTextFiles

I am trying to use mapPartitions and then save the results to HDFS by the following:

val x = sc.parallelize(List(1, 2, 3, 4, 5, 6, 7, 8, 9,10), 3)
x.mapPartitions(p =>  p.map( r => r+5 ) ).collect().saveAsTextFile("/path/to/folder")

:29: error: value saveAsTextFile is not a member of Array[Int] x.mapPartitions(p => p.map( r => r+5 ) ).collect().saveAsTextFile("/path/to/folder")

It seems something is wrong but I failed to figure out what is the correct way.

THanks

Upvotes: 2

Views: 278

Answers (1)

user8122169
user8122169

Reputation: 46

Don't use collect:

x.mapPartitions(p =>  p.map( r => r+5 ) ).saveAsTextFile("/path/to/folder")

It collects data to the driver as an Array.

Upvotes: 3

Related Questions