CODEWITHSUNDEEP
csizempirankprocessors
Arjun Singh
Arjun Singh

Reputation: 29

Calculate valid range of values for the variables rank and size for MPI program Intel i7 8th Generation (6 cores with 2 threads per core)

Please suggest how to calculate valid rank and size for mpi:

The Hello World C-MPI program below is to be compiled onto an executable file named ./hello and executed on a server with an Intel i7 8th Generation (6 cores with 2 threads per core) and 16GB of memory with Linux Ubuntu 16 and the full MPI library.

#include <mpi.h>
#include <stdio.h>
int main (int argc, char* argv[])
{
int rank, size;
MPI_Init (&argc, &argv); /* starts MPI */
MPI_Comm_rank (MPI_COMM_WORLD, &rank); /* get current process id */
MPI_Comm_size (MPI_COMM_WORLD, &size); /* get number of processes */
printf( "Hello world from process %d of %d\n", rank, size );
MPI_Finalize();
return 0;
}

Is the mpirun -np 24 ./hello a valid execution command? Justify your answer and, if your answer is positive, provide the valid range of values for the variables rank and size.

Upvotes: 0

Views: 463

Answers (1)

Hristo Iliev
Hristo Iliev

Reputation: 74475

This is an ill-posed question since the maximum number of ranks depends on so many factors that it is hard to give a short and/or complete answer.

First of all, the output arguments of both MPI_Comm_size() and MPI_Comm_rank() are int, which is 32-bit on most platforms nowadays, therefore imposing a hard upper limit of 2 billion (231-1) ranks.

Then comes the limit that the particular MPI implementation imposes. Unfortunately, MPI does not specify a way to obtain the maximum number of ranks in an MPI job and you have to consult the implementation's documentation, but let's say most use ints for internal indexing, so the limit is probably the same - 2 billion ranks.

Then comes the question of how exactly the MPI library implements MPI ranks. With most off-the-shelf MPIs on Linux that would be OS processes, therefore we need to look into how many processes can be run simultaneously.

First, there is the maximum number of processes that the Linux kernel can handle. By default, the process ID (PID) wraps around when it reaches the value specified in /proc/sys/kernel/pid_max, which is 32768 when the system boots. That's the upper limit on 32-bit Linux and also the default limit on 64-bit Linux (for backward compatibility), but it can be changed up to 222 on 64-bit Linux. The PIDs for user processes start at 300 except for the very special first user process (the venerable init) that is always 1. So you can have up to 222-300 simultaneously running processes besides init, about 4,2 million, but it's usually less since kernel threads can eat into that PID space too.

Then there are the user limits imposed by the ulimit mechanism. ulimit -u sets or obtains the limit of running processes for the user. The default on Ubuntu 18.04 is 483294, which is also the maximum possible value for that limit (i.e., the hard limit). Since MPI jobs typically run under the same username, this limit caps the number of ranks one can have.

Typical MPI implementations come with a special launcher process, therefore you cannot launch an MPI job with more than 483293 ranks on 64-bit Ubuntu 18.04. But again, that's just an upper limit. There's more.

OS processes each have their own virtual address space page tables and corresponding kernel structures and those consume memory. On top of that are any additional memory allocations done by the process. The minimal MPI program from your question with an added call to sleep(3) in order to give one the time to run the ps command before the program exits, compiled with Open MPI on 64-bit Ubuntu 18.04 and stripped of debugging symbols, has a resident set size (RSS) of 12 MiB per rank. The mpiexec launcher has a bit more and its memory usage grows with the number of MPI ranks, but let's ignore that. Ignoring also the kernel memory use and the fact that there are other processes on a typical Linux system, you cannot run more than (16 GiB + amount of swap) / 12 MiB processes. Swap size typically ranges from no swap at all to double the RAM size, although it could be really anything. Given no swap, you are limited to 1364 ranks. With swap size twice the size of RAM, you can have 4095 ranks. Let's assume for simplicity the former case, i.e., no swap, therefore a limit of 1364 ranks.

The simple program does nothing computationally intensive. If it was actually computing something, then those 1364 ranks would be competing for 12 logical CPUs, for 114 processes per local CPU, which is a bit absurd. Therefore, a more sensible limit would be 12 if there are good options for memory latency hiding by utilising the hyperthreading, or even 6 only, if the program is running some very tight loop that don't constantly reach for data outside the core-private cache.

So, as you can see, your question about the valid range is ill-posed. As for running on 12 logical CPUs a total of 24 MPI ranks that simply output "Hello world", that's fine.

As for what the values of rank and size could be when you start a 24-rank MPI job, the answer is found in the fact that the size of MPI_COMM_WORLD is fixed globally for the MPI job and is that of the number of MPI ranks requested initially with the -np parameter, while the actual rank number varies from 0 to size - 1.

Upvotes: 1

Related Questions