CODEWITHSUNDEEP
pythonapache-sparkpysparkparquetapache-spark-2.0
saloua
saloua

Reputation: 2493

Merging schemas when reading parquet files fails because of incompatible data types int and bigint

When trying to load parquet files with schema merge

   df = spark.read.option("mergeSchema", "true").parquet('some_path/partition_date')
   df.show()

I'm getting the following exception:

    Py4JJavaError: An error occurred while calling o421.parquet.
: org.apache.spark.SparkException: Failed merging schema:
root
....
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$mergeSchemasInParallel$1.apply(ParquetFileFormat.scala:643)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$mergeSchemasInParallel$1.apply(ParquetFileFormat.scala:639)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.mergeSchemasInParallel(ParquetFileFormat.scala:639)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.inferSchema(ParquetFileFormat.scala:241)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource.scala:180)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource.scala:180)
    at scala.Option.orElse(Option.scala:289)
    at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:179)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:373)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
    at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:643)
    at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Failed to merge fields 'some_field_name' and 'some_field_name'. Failed to merge incompatible data types int and bigint

Is there a better way to read the data having some schema evolution including incompatible types? Thanks

Upvotes: 6

Views: 5448

Answers (1)

Nastasia
Nastasia

Reputation: 657

You could load each different files separately by forcing the schema and/or using casts. Then, you can do a union.

Upvotes: 2

Related Questions