CODEWITHSUNDEEP
javaapache-sparkapache-spark-sqlhdfs
Alperen Kezay
Alperen Kezay

Reputation: 3

java.lang.NoClassDefFoundError: org/apache/spark/TaskOutputFileAlreadyExistException

I have read the data in HDFS. I analyzed it, but I get this error when writing. Continuation of the error

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/TaskOutputFileAlreadyExistException
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:167)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:123)
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:173)
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:211)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:208)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:169)
    at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:110)
    at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:109)
    at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:828)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$4(SQLExecution.scala:100)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:87)
    at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:828)
    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:309)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:236)
    at SparkSQL.SparkHDFS.main(SparkHDFS.java:22)

My code

SparkSession sparkSession = SparkSession.builder().appName("FirstSQL").master("local").getOrCreate();

Encoder<MovieModal> movieModalEncoder = Encoders.bean(MovieModal.class);

Dataset<MovieModal> data = sparkSession.read().option("infershema",true)
                                        .option("header",true)
                                        .csv("hdfs://localhost:8020/data/ratings.csv")
                                        .as(movieModalEncoder);


Dataset<Row> groupData = data.groupBy(new Column("movieID")).count();

groupData.write().format("csv").save("hdfs://localhost:8020/var/groupData2.csv");

Upvotes: 0

Views: 1132

Answers (1)

notNull
notNull

Reputation: 31540

If directory already exists then we need to provide either overwrite(overwrites existing directory) or append(appends to the directory) as mode while writing.

Try with:

groupData.write().mode("overwrite").format("csv").save("hdfs://localhost:8020/var/groupData2.csv");

Upvotes: 1

Related Questions