CODEWITHSUNDEEP
apache-sparkapache-spark-ml
Luckylukee
Luckylukee

Reputation: 595

Getting elements from the output of vectorassembler

I need to get the output elements of a vector assembler as separate columns using the Java API.

VectorAssembler assembler3 = new VectorAssembler()
              .setInputCols(new String[]{"res1", "res2"})
              .setOutputCol("res3");

DataFrame output = assembler1.transform(sensordataDF);

res1 and res2 are both double array vectors. Can anybody guide me how to do this?

Upvotes: 0

Views: 1083

Answers (1)

Luis A.G.
Luis A.G.

Reputation: 1097

The output dataframe will be sensordataDF with a new column called res3, but also it will still have columns res1 and res2.

Edit: Maybe could be done using spark.sql.functions split and casting the column to string, and then while separating, casting back to doubletype.

I use spark with python, but in java should be nearly the same

Example:

split_col = split(output['res3'], ',')

df = ouput.withColumn('first_data', split_col.getItem(0))
df = df.withColumn('second_data', split_col.getItem(1))

Upvotes: 1

Related Questions