CODEWITHSUNDEEP
pyspark
Erin
Erin

Reputation: 505

PySpark transform method with Vector Assembler

I have a Spark DataFrame and I would like to use Vector Assembler to create a "features" column.

from pyspark.ml.feature import VectorAssembler

assembler = VectorAssembler(inputCols=sel_cols, outputCol='features')

transformed_data = assembler.transform(sdf)

sel_cols consists of a list of 150 items of string type, which looks as follows:

['ASP.NET Core',
 'ASP.NET MVC',
 'AWS',
 'AWS DynamoDB',
 'AWS EMR',
 'AWS SDK',
 'Adobe Photoshop',
 'Agile',
 'Agile software development29',
 'Ajax',
 'Amazon CloudFront CDN',
 'Amazon EC2',
 'Android',
 'Angular',
 'AngularJS',
 'Apache',
 'Apache Hive',
 'Apache Spark',
 'Atom',...]

And sdf.columns, consists of 340 items and looks as follows:

['.NET',
 '.NET 4',
 '.NET Core',
 'ADO.NET',
 'AFNetworking',
 'API Architecture',
 'API Design',
 'API Development',
 'APIs',
 'ASP.NET',
 'ASP.NET Core',
 'ASP.NET MVC',
 'ASP.NET Web API',
 'AWS',
 'AWS DynamoDB',...]

I am getting this error in applying transformed_data = assembler.transform(sdf):

AnalysisException: Cannot resolve column name "ASP.NET Core" among (.NET, .NET 4, .NET Core, ADO.NET, AFNetworking, API Architecture, API Design, API Development, APIs, ASP.NET, ASP.NET Core, ASP.NET MVC, ASP.NET Web API, AWS, AWS DynamoDB, AWS EC2, AWS ECS, AWS EMR, AWS HA, AWS Lambda, AWS RDS, AWS S3, AWS SDK, Adobe Illustrator,...

As shown, "ASP.NET Core" is definitely among my sdf.columns and as far as I understand it, passing sel_cols as a list of string to VectorAssembler's inputCols should work... Would really appreciate any insight as I haven't worked with Spark DF's before:)

Thank you!

Upvotes: 1

Views: 1626

Answers (1)

werner
werner

Reputation: 14845

The VectorAssembler cannot handle columns with a space or a dot in the column name. Another answer from me provides some technical background why it does not work.

The only option is to rename the columns:

#build a dict (orginal column name: new column name)
mapping = {col: col.replace('.','_').replace(' ', '_') for col in sel_columns}

#select all columns and create an alias if there is a mapping for this column
df_renamed = df.select([F.col('`'+c+'`').alias(mapping.get(c, c)) for c in df.columns])

#create a VectorAssembler that uses the renamed columns as input
assembler = VectorAssembler(inputCols=list(mapping.values()), outputCol='features')
transformed_data = assembler.transform(df_renamed)

Upvotes: 1

Related Questions