CODEWITHSUNDEEP
scalaapache-spark-sqlunion
Erkan Şirin
Erkan Şirin

Reputation: 2085

Spark Scala union fails although two dataframes have the same schema

On Windows, Spark 2.3.1 I try to union two dataframes. Although both have the same schema I get an error saying "Union can only be performed on tables with the compatible column types" I don't understand why. Because I have done the second transformation in order to get the desired schema for the second dataframe.

My codes:

import breeze.linalg._
import org.apache.spark.ml.feature.{StandardScaler, VectorAssembler}
import org.apache.spark.mllib.linalg.{Vector, Vectors}
import org.apache.spark.mllib.stat.Statistics
import org.apache.spark.sql.{SparkSession}
import org.apache.spark.sql.functions.{rand => random, udf,col}
import org.apache.spark.sql.types._

object MahalanobisDeneme {
  def main(args: Array[String]): Unit = {

    val spark = SparkSession.builder()
      .appName("KMeansZScore")
      .master("local[*]")
      .getOrCreate()
    import spark.implicits._

    val df = spark.range(0, 10).select("id").
      withColumn("uniform", random(10L)).
      withColumn("normal1", random(10L)).
      withColumn("normal2", random(11L))


   //df.show()

    val assembler = new VectorAssembler()
      .setInputCols(Array("uniform", "normal1", "normal2"))
      .setOutputCol("features")
    val assembledDF = assembler.transform(df)
    //assembledDF.show()


    val idFeaturesDF = assembledDF.select("id","features")
    idFeaturesDF.show(false)
    idFeaturesDF.printSchema()


    val outlierDF = spark.createDataFrame(Seq((10, Vectors.dense(5,5,5))))

    val outlierDF2 = outlierDF
      .withColumn("id", outlierDF.col("_1").cast("Long"))
        .withColumn("features",outlierDF.col("_2"))
    .select("id","features")

      outlierDF2.show()
    outlierDF2.printSchema()

    val unionedDF = idFeaturesDF.union(outlierDF2)
    unionedDF.show()
}
}

Schema output for idFeaturesDF:

root
 |-- id: long (nullable = false)
 |-- features: vector (nullable = true)

Schema output for outlierDF2:

root
 |-- id: long (nullable = false)
 |-- features: vector (nullable = true)

More error logs are below:

  Exception in thread "main" org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. 
struct<type:tinyint,size:int,indices:array<int>,values:array<double>> <> 
struct<type:tinyint,size:int,indices:array<int>,values:array<double>> at the second column of the second table;;
    'Union
    :- AnalysisBarrier
    :     +- Project [id#0L, features#15]
    :        +- Project [id#0L, uniform#3, normal1#6, normal2#10, UDF(named_struct(uniform, uniform#3, normal1, normal1#6, normal2, normal2#10)) AS features#15]
    :           +- Project [id#0L, uniform#3, normal1#6, rand(11) AS normal2#10]
    :              +- Project [id#0L, uniform#3, rand(10) AS normal1#6]
    :                 +- Project [id#0L, rand(10) AS uniform#3]
    :                    +- Project [id#0L]
    :                       +- Range (0, 10, step=1, splits=Some(8))
    +- AnalysisBarrier
          +- Project [id#35L, features#39]
             +- Project [_1#31, _2#32, id#35L, _2#32 AS features#39]
                +- Project [_1#31, _2#32, cast(_1#31 as bigint) AS id#35L]
                   +- LocalRelation [_1#31, _2#32]

Upvotes: 1

Views: 1855

Answers (1)

Joe Pallas
Joe Pallas

Reputation: 2155

Try changing import org.apache.spark.mllib.linalg.{Vector, Vectors} to import org.apache.spark.ml.linalg.{Vector, Vectors}. They look the same to printSchema() but they are different.

(I compared idFeaturesDF.head.get(1).getClass and outlierDF2.head.get(1).getClass because the message complained about the second column.)

Upvotes: 2

Related Questions