CODEWITHSUNDEEP
pythonnumpyapache-sparkpyspark
pauli
pauli

Reputation: 4291

Difference between math and numpy functions in pyspark

Why pyspark behaves differently when numpy functions are used instead of math functions? For example

X = sc.parallelize([[DenseVector([4.9, 3.0, 1.4, 0.2]), DenseVector([4.6, 3.1, 1.5, 0.2])],[DenseVector([5.1, 3.5, 1.4, 0.3]), DenseVector([5.7, 3.8, 1.7, 0.3])]])
X_df = sqlcontext.createDataFrame(X, ["x","y"])
udf_foo = udf(lambda x, y:  m.exp(-x.squared_distance(y)/2.0), DoubleType())
X_sim = X_df.withColumn("sim", udf_foo(X_df.x, X_df.y))

X_sim.show()

output

+-----------------+-----------------+------------------+
|                x|                y|               sim|
+-----------------+-----------------+------------------+
|[4.9,3.0,1.4,0.2]|[4.6,3.1,1.5,0.2]|0.9464851479534836|
|[5.1,3.5,1.4,0.3]|[5.7,3.8,1.7,0.3]|0.7633794943368529|
+-----------------+-----------------+------------------+

while code below 

udf_foonp = udf(lambda x, y:  np.exp(-x.squared_distance(y)/2.0), DoubleType())
X_simnp = X_df.withColumn("sim", udf_foonp(X_df.x, X_df.y))

X_simnp.show()

gives an error 

expected zero arguments for construction of ClassDict

Upvotes: 2

Views: 648

Answers (1)

zero323
zero323

Reputation: 330093

Because return type is different:

type(np.exp(1.0))
## numpy.float64

type(math.exp(1.0))
## float

and NumPy types are not valid external representation for the SQL types. Therefore you have to cast:

udf(lambda x, y: float(np.exp(-x.squared_distance(y) / 2.0)), DoubleType())

Upvotes: 1

Related Questions