CODEWITHSUNDEEP
apache-spark-sqlpyspark
Jge
Jge

Reputation: 165

Error when trying to specify schema for loading a CSV using pyspark

I am trying to build a schema to load the contents of a file which contains the following things-

  1. movie_id, String
  2. movie_name, String
  3. plot, String
  4. genre, an array of strings

Here is a sample- enter image description here

Here is my schema definition-

customSchema = types.StructType(types.ArrayType(
                         types.StructField("movie_id", types.StringType(), True),
                         types.StructField("movie_name", types.StringType(), True),
                         types.StructField("plot", types.StringType(), True),
                         types.StructField('genre', 
                         types.ArrayType(types.StringType()), True),
))

Here is the error that I am getting

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-290-10452c69a6ff> in <module>()
        3   types.StructField("movie_name", types.StringType(), True),
        4   types.StructField("plot", types.StringType(), True),
  ----> 5   types.StructField('genre', types.ArrayType(types.StringType()), True),
        6 ))

TypeError: __init__() takes from 2 to 3 positional arguments but 5 were given

Upvotes: 2

Views: 573

Answers (1)

murtihash
murtihash

Reputation: 8410

StructType() is a list of StructField()s.

from pyspark.sql.types import *  

customSchema= StructType([StructField("movie_id", IntegerType()),
                        StructField("movie_name", StringType()),
                        StructField("plot", StringType()),
                          StructField("genre", ArrayType(StringType()))])

If genre string column looks like "['Indie', 'Drama', 'Action']", try this to convert to array of strings:

from pyspark.sql import functions as F
df.withColumn("genre", F.split(F.regexp_replace("genre", "\[|]| |'", ""),",")).show(truncate=False)

#+----------------------+
#|genre                 |
#+----------------------+
#|[Indie, Drama, Action]|
#+----------------------+

Upvotes: 1

Related Questions