CODEWITHSUNDEEP
apache-spark
Saurabh Santhosh
Saurabh Santhosh

Reputation: 130

java.lang.NoSuchMethodError: parquet.Preconditions.checkState(ZLjava/lang/String;)V

Got the following error while running Spark 1.3.1 as well as 1.4.1

java.lang.NoSuchMethodError: parquet.Preconditions.checkState(ZLjava/lang/String;)V at parquet.schema.Types$PrimitiveBuilder.build(Types.java:314) at parquet.schema.Types$PrimitiveBuilder.build(Types.java:232) at parquet.schema.Types$Builder.named(Types.java:210) at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$fromDataType$1.apply(ParquetTypes.scala:314) at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$fromDataType$1.apply(ParquetTypes.scala:305) at scala.Option.map(Option.scala:145) at org.apache.spark.sql.parquet.ParquetTypesConverter$.fromDataType(ParquetTypes.scala:305) at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$4.apply(ParquetTypes.scala:395) at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$4.apply(ParquetTypes.scala:394) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.parquet.ParquetTypesConverter$.convertFromAttributes(ParquetTypes.scala:393) at org.apache.spark.sql.parquet.ParquetTypesConverter$.writeMetaData(ParquetTypes.scala:440) at org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache.prepareMetadata(newParquet.scala:260) at org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:276) at org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:269) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache.refresh(newParquet.scala:269) at org.apache.spark.sql.parquet.ParquetRelation2.(newParquet.scala:391) at org.apache.spark.sql.parquet.DefaultSource.createRelation(newParquet.scala:98) at org.apache.spark.sql.parquet.DefaultSource.createRelation(newParquet.scala:128) at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:240) at org.apache.spark.sql.hive.execution.CreateMetastoreDataSourceAsSelect.run(commands.scala:218) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:54) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:54) at org.apache.spark.sql.execution.ExecutedCommand.execute(commands.scala:64) at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:1099) at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:1099) at org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1121) at org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1071) at org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1037) at org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1015) at com.xurmo.ai.spark.SparkClient.dataFrameToHiveTable(SparkClient.java:128) at com.xurmo.ai.xflow.operation.sink.DataFrameToPlatformSink.push(DataFrameToPlatformSink.java:79) at com.xurmo.ai.xflow.operation.sink.ASink.operate(ASink.java:24) at com.xurmo.ai.xflow.operation.AOperation.process(AOperation.java:121) at com.xurmo.ai.xflow.flow.executor.ExecutableOp.call(ExecutableOp.java:26) at com.xurmo.ai.xflow.flow.executor.ExecutableOp.call(ExecutableOp.java:15) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722)

From the log its clear that the class loader is loading some other version of this class. Can someone please tell me which other jar packages have this class?

Upvotes: 2

Views: 2251

Answers (1)

Gabber
Gabber

Reputation: 7249

I had a same issue. There are two jar file in lib directory which contain Precondition Class

1.parquet-hadoop-bundle-1.6.0rc3.jar
2.parequet-hadooop-bundle-***.jar (which does not contain checkState method)

I remove it only 

parequet-hadooop-bundle-***.jar

file from lib directory and now its working for me.

Upvotes: 2

Related Questions