CODEWITHSUNDEEP
apache-sparktypespyspark
Azamat
Azamat

Reputation: 249

TypeError while manipulating arrays in pyspark

I am trying to compute dot product (sum of element products) between 'user_features' and 'movie_features':

+------+-------+--------------------+--------------------+
|userId|movieId|       user_features|      movie_features|
+------+-------+--------------------+--------------------+
|    18|      1|[0.0, 0.5, 0.0, 0...|[1, 0, 0, 0, 0, 1...|
|    18|      2|[0.1, 0.0, 0.0, 0...|[1, 0, 0, 0, 0, 0...|
|    18|      3|[0.2, 0.0, 0.3, 0...|[0, 0, 0, 0, 0, 1...|
|    18|      4|[0.0, 0.1, 0.0, 0...|[0, 0, 0, 0, 0, 1...|
+------+-------+--------------------+--------------------+

Data types:

df.printSchema()
_____________________________________________
root
 |-- userId: integer (nullable = true)
 |-- movieId: integer (nullable = true)
 |-- user_features: array (nullable = false)
 |    |-- element: double (containsNull = true)
 |-- movie_features: array (nullable = false)
 |    |-- element: float (containsNull = true)

None

I use this

class Solution:
    """
    Data reading, pre-processing...
    """
    @udf("array<double>")
    def miltiply(self, x, y):
        if x and y:
            return [float(a * b) for a, b in zip(x, y)]
    
    def get_dot_product(self):
        
        df = self.user_DF.crossJoin(self.movies_DF)
        output = df.withColumn("zipxy", self.miltiply("user_features", "movie_features")) \
                   .withColumn('sumxy', sum([F.col('zipxy').getItem(i) for i in range(20)]))

Gives the following error:

TypeError: Invalid argument, not a string or column: <__main__.Solution instance at 0x000000000A777EC8> of type <type 'instance'>. For column literals, use 'lit', 'array', 'struct' or 'create_map' function.

What am I missing? I am doing it by udf since I am using Spark 1.6 therefor can't use aggregate or zip_with functions.

Upvotes: 0

Views: 75

Answers (1)

Lamanus
Lamanus

Reputation: 13581

If you can use the numpy then

df = spark.createDataFrame([(18, 1, [1, 0, 1], [1, 1, 1])]).toDF('userId','movieId','user_features','movie_features')

import numpy as np
df.rdd.map(lambda x: (x[0], x[1], x[2], x[3], float(np.dot(np.array(x[2]), np.array(x[3]))))).toDF(df.columns + ['dot']).show()

+------+-------+-------------+--------------+---+
|userId|movieId|user_features|movie_features|dot|
+------+-------+-------------+--------------+---+
|    18|      1|    [1, 0, 1]|     [1, 1, 1]|2.0|
+------+-------+-------------+--------------+---+

Upvotes: 1

Related Questions