CODEWITHSUNDEEP
python-3.xdataframepysparkavrospark-avro
T.SURESH ARUNACHALAM
T.SURESH ARUNACHALAM

Reputation: 285

How to append data in existing AVRO file using Python

I have a dataframe with similar schema, I need to append the data into the AVRO file. I don't like to add the avro file into folder as a part. For your information, my AVRO file is not into the folder as a part. Can you please help me to solve the task.

Upvotes: 1

Views: 774

Answers (1)

Shubham Jain
Shubham Jain

Reputation: 5526

You can write the data by using mode overwrite while writing the dataframe.

But the part file is created as spark is distributed processing and each executor spits out a files based on the amount of data

Upvotes: 0

Related Questions