CODEWITHSUNDEEP
amazon-web-servicesamazon-s3pysparkaws-glue
Eumcoz
Eumcoz

Reputation: 2458

Glue Job fails to write file

I am back filling some data via glue jobs. The job itself is reading in a TSV from s3, transforming the data slightly, and writing it in Parquet to S3. Since I already have the data, I am trying to launch multiple jobs at once to reduce the amount of time needed to process it all. When I launch multiple jobs at the same time, I run into an issue sometimes where one of the files will fail to output the resultant Parquet files in S3. The job itself completes successfully without throwing an error When I rerun the job as a non-parallel task, the file it output correctly. Is there some issue, either with glue(or the underlying spark) or S3 that would cause my issue?

Upvotes: 2

Views: 1361

Answers (1)

Yuriy Bondaruk
Yuriy Bondaruk

Reputation: 4750

The same Glue job running in parallel may produce files with the same names and therefore some of them can be overwritten. As I remember correctly, transformation-context is used as part of the name. I assume you don't have bookmarking enabled so it should be safe for you to generate transformation-context value dynamically to ensure it's unique for each job.

Upvotes: 2

Related Questions