CODEWITHSUNDEEP
pythonslurm
paulo
paulo

Reputation: 65

Python job on slurm cluster, nodes vs cores

I have an extremely basic question, to which I somehow never managed to find an answer. Let's assume that I have access to a cluster running slurm, and that I need to run a Python job on the cluster. Let us assume that my code has not been written to support multiprocessing. Do I have any reason to require multiple cores? Or should I then stick to 1 node and 1 core?

Conversely, if I wanted to run 5 times the same script (with different input variables for example), is there any difference between requiring 1 node and assigning 1 core to each job, or requiring 5 nodes with 1 core each?

Upvotes: 2

Views: 530

Answers (1)

Marcus Boden
Marcus Boden

Reputation: 1685

If your python script does not use multiple threads, then yes, you should stick to one task (-n1) with one CPU (-c1) on one node (-N1). You don't need to specify that as it is the default anyway. If you request more resources, they will just be wasted, as you don't use them.

However: Some python libraries to multithreaded calculations without the need to specify them explicitly, so if you do some numpy calculations, you may benefit from multiple cores.

Conversely, if I wanted to run 5 times the same script (with different input variables for example), is there any difference between requiring 1 node and assigning 1 core to each job, or requiring 5 nodes with 1 core each?

Yes, there is: If you ask Slurm for 5 Nodes with 1 task per node, then it will have to wait until 5 nodes have room for a task. Even if for example a node with 20 CPUs is completely empty, your job won't run, as you explicitly asked for 5 nodes. So I would advise to start 5 jobs with -n1.

Upvotes: 1

Related Questions