CODEWITHSUNDEEP
aws-glue
Yuva
Yuva

Reputation: 3163

AWS Glue DPU configurations

I see that the DPU is made of 4 vCPUs & 16 GB memory. Is it possible to change this settings for vCPU, memory, so that I don't run out of DPUs or exceed the DPU limit.

I think there is a maximum limit of 5 DPUs for a Dev Endpoint, and a maximum of 2 DEV Endpoints for an account?

Regards

Yuva

Upvotes: 2

Views: 5669

Answers (2)

Sinan Erdem
Sinan Erdem

Reputation: 1124

As of April 2019, there are two new types of workers:

You can now specify a worker type for Apache Spark jobs in AWS Glue for memory intensive workloads.

Previously, all Apache Spark jobs in AWS Glue ran with a standard configuration of 1 Data Processing Unit (DPU) per worker node and 2 Apache Spark executors per node. You can now pick from two new configurations, G.1X and G.2X, that provide more memory per executor.

https://aws.amazon.com/about-aws/whats-new/2019/04/aws-glue-now-supports-additional-configuration-options-for-memory-intensive-jobs/

Upvotes: 1

charmander
charmander

Reputation: 1085

Right now there is no way to configure DPU memory, but you can request a limit increase on your account to be able to use more DPUs.

Upvotes: 3

Related Questions