can't pickle listreverseiterator objects using PySpark

Question

I'm quite new to python. The following code has some serialization problems:

from pyspark.mllib.stat import Statistics
import random
data = sc.parallelize([range(100),range(100),random.sample(range(100),100),reversed(range(100))])
print(Statistics.corr(data))

I'm getting:

TypeError: can't pickle listreverseiterator objects

So "reversed" seems to create a non-serializeable object - any idea how to cope with this?

thanks a lot!

Answer 1

Replace:

reversed(range(100))

with

list(reversed(range(100)))