CODEWITHSUNDEEP
apache-spark-sqlspark-avroapache-spark-1.6
Mnav505
Mnav505

Reputation: 13

Spark 1.6 - Overwrite directory with avro files failing using dataframes

I have a directory in HDFS which contains avro files. While I try to overwrite the directory with dataframe it fails. Syntax: avroData_df.write.mode(SaveMode.Overwrite).format("com.databricks.spark.avro").save("") The error is: Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://nameservice1//part-r-00000-bca9a5b6-5e12-45c1-a877-b0f6d6cc8cd3.avro

It somehow seems to be considering the avro files as well while overwriting.

Can we do this using Spark 1.6 at all?

Upvotes: 1

Views: 182

Answers (0)

Related Questions