CODEWITHSUNDEEP
apache-sparkapache-spark-sqlspark-javalz4
ujjawal
ujjawal

Reputation: 1

java.lang.NoSuchFieldError: LZ4 : spark.sql()

I am trying to run one of my methods which has -

spark.sql("DROP TABLE IF EXISTS " + dbNameString + "." + tableNameString)

When I am running the method, the code breaks on the above line throwing following stacktrace -

java.lang.NoSuchFieldError: LZ4

    at org.apache.spark.sql.execution.datasources.parquet.ParquetOptions$.<init>(ParquetOptions.scala:97)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetOptions$.<clinit>(ParquetOptions.scala)
    at org.apache.spark.sql.hive.HiveMetastoreCatalog.convert(HiveMetastoreCatalog.scala:136)

I am using Java 8 and following -

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.12</artifactId>
    <version>3.4.1</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.12</artifactId>
    <version>3.4.1</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-hive_2.12</artifactId>
    <version>3.4.1</version>
</dependency>
<dependency>
      <groupId>org.apache.hive</groupId>
      <artifactId>hive-metastore</artifactId>
      <version>2.3.9</version>
      <scope>provided</scope>
</dependency>

I am assuming that this might be a JAR conflict and tried checking lz4 dependencies but nothing seems to be conflicting in terms of lz4.

<dependency>
        <groupId>org.lz4</groupId>
        <artifactId>lz4-java</artifactId>
        <version>1.7.1</version>
</dependency>

I tried versioning up lz4-java to 1.8.0 but that also didn't work.

Upvotes: 0

Views: 102

Answers (0)

Related Questions