CODEWITHSUNDEEP
pythondataframepysparkaws-gluedelta
Rahul I
Rahul I

Reputation: 11

Writing incorrect data into delta format file from data frame in AWS glue job

Our Glue job is writing data into delta format file from data frame. It looks like data frame is fetching correct records however after writing into delta format , the data is in consistent(sometimes writes the correct record and sometimes writes incorrect random record). Need some help to resolve this

Here is the code snippet I used to write into delta format file :

additional_options = {
            "path": fld_path,
            "write.parquet.compression-codec": "snappy"        
        }
(filtered_df2.write.format("delta")
                         .options(**additional_options)
                         .mode("overwrite")
                         .save())

Thanks in Advance

Upvotes: 1

Views: 27

Answers (0)

Related Questions