CODEWITHSUNDEEP
amazon-web-servicesaws-glue
Melissa Guo
Melissa Guo

Reputation: 1088

How do I handle a no space left error in AWS Glue DataBrew?

When running for 1 month's worth of data I get the below errors. In the advanced settings I have the maximum number of units set to the maximum of 10. What is the recommended approach to handle this?

An error occurred while calling o62075.collectToPython. No space left on device

An error occurred while calling o62081.pyWriteDynamicFrame. /tmp/blockmgr-9adbd7f0-0068-4244-bf85-3c59df19b610/33/temp_shuffle_4ec99ff1-e730-46ab-8b94-e35d51308aa6 (No space left on device)

Upvotes: 2

Views: 739

Answers (1)

Nitya Sheth
Nitya Sheth

Reputation: 160

I think you are in the right direction about the units. Depending on your dataset size, the recommended approach is to increase the number of units for a given job. For all subsequent job runs, DataBrew allocate that number of units for processing.

Maximum you can allocate is 149 units. Link to official docs - here

Upvotes: 1

Related Questions