CODEWITHSUNDEEP
jobsslurm
khaverim
khaverim

Reputation: 3554

Is it possible to request more time to a running job in SLURM?

I know it's possible on a queued job to change directives via scontrol, for example

scontrol update jobid=111111 TimeLimit=08:00:00

This only works in some cases, depending on the administrative configuration of the slurm instance (I'm not an admin). Thus this post does not answer my question.

What I'm looking for is a way to ask SLURM to add more time to a running job, if resources are available, and even if it's already running. Sort of like a nested job request.

Particularly a running job that was initiated with srun on-the-fly.

Upvotes: 1

Views: 5557

Answers (1)

Chelmy88
Chelmy88

Reputation: 1116

In https://slurm.schedmd.com/scontrol.html, it is clearly written under TimeLimit:

Only the Slurm administrator or root can increase job's TimeLimit.

So I fear what you want is not possible.

An it makes sense, since the scheduler looks at job time to decide which jobs to launch and some short jobs can benefit from back-filling to start before longer jobs, it would be really a mess if users where allowed to change the job length while running. Indeed, how to define "when resource are available"? Some node can be IDLE for some time because slurm knows that it will need it soon for a large job

Upvotes: 3

Related Questions