CODEWITHSUNDEEP
amazon-web-servicesaws-batch
Ben
Ben

Reputation: 1342

why wont my aws batch job all start and run in paralllel?

hope someone can help me stop tearing my hair out!

I have a a job with array of ~700 indexes

When i submit the job , I get no more than 20-30 running simultaneously They all run eventually which leads me to assume its a constraint else where and as all jobs the same, its not permissions/roles/connectivity.

They are array / index jobs, and one job in the queue I can't find any limits on these types of jobs running?

note i'm using ec2 unmanaged as the job was too big for fargate

i've tried

  1. double checked they are parallel not sequential
  2. dropped individual cpu / membory for each job to 0.25vcpu and 1gb memmory
  3. created 'huge' compute environments of max 4096 vpu - no desired or min
  4. added upto 3 compute env to a queue (as per limit)

what am i missing? hope someone can point me in a different direction thanks Ben

Upvotes: 1

Views: 1241

Answers (1)

Marcin
Marcin

Reputation: 238747

Based on the comments.

The issue was caused by EC2 service limits. AWS Bash will use EC2 to run the jobs, and it will not launch more resources then those specified by the EC2 limits. You can request increase the service quota of my Amazon EC2 resources to overcome the issue.

Upvotes: 2

Related Questions