CODEWITHSUNDEEP
c++linuxmultithreadingprocessopenmp
ricky116
ricky116

Reputation: 774

Problems spawning a multi-threaded process from a separate single-threaded process in Linux

I've had some rather unusual behavior from my OpenMP program, which I think must be caused by some inner-workings of Linux processes that I am unaware of.

When I run a C benchmark binary which has been compiled with OpenMP support, the program executes successfully, no problems at all:

>:$ OMP_NUM_THREADS=4 GOMP_CPU_AFFINITY="0-3" /home/me/bin/benchmark
>:$ ...benchmark complete...

When I run the benchmark from a separate C++ program that I start, where the code to execute it (for example) looks like:

int main(int argc, char* argv[]){

    system("/home/me/bin/benchmark");

    return 0;
}

The output gives me warnings:

>:$ home/me/bin/my_cpp_program
OMP: Warning #123: Ignoring invalid OS proc ID 1.
OMP: Warning #123: Ignoring invalid OS proc ID 2.
OMP: Warning #123: Ignoring invalid OS proc ID 3.

These warnings are the same warnings I get when I try to set CPU affinity to CPUs that don't exist, and run the OpenMP benchmark directly.

I therefore assume that the only CPU my_cpp_program knows to exist is processor ID 0. I also get the same error when using root so I don't think it is a permissions problem? I've also checked that the code executed by system() has the correct environment variables, and the same linked libraries.

Does anyone know what is causing this to occur?

Upvotes: 0

Views: 433

Answers (1)

ricky116
ricky116

Reputation: 774

Following a suggestion by Zulan, I'm pretty sure the reason this occurred was because of the nature and compilation of bin/my_cpp_program. It was importantly also compiled with -fopenmp. It turns out that bin/my_cpp_program was compiled against the GNU OpenMP implementation library libgomp, whereas the bin/benchmark program was compiled against the LLVM OpenMP implementation library libomp.

I am not certain, but what I think happened was the following. GOMP_CPU_AFFINITIY is the GNU environment variable for setting CPU affinity in libgomp. Therefore because GOMP_CPU_AFFINITY was set, the single-thread running in bin/my_cpp_program was bound to CPU 0. I guess any child processes spawned by that thread must also only see CPU 0 as its potential-CPUs when using further GOMP_CPU_AFFINITY affinity assignments. This then gave me the warnings when the spawned process (from the environment) tries to find CPUs 1-3.

To workaround this I used KMP_AFFINITY which is Intel's CPU affinity environment variable. The libomp OpenMP runtime used by bin/benchmark gives a higher precedence to KMP_AFFINITY when both it and GOMP_CPU_AFFINITY are set, and for whatever reason this allows the spawned child process to assign to other CPUs. So to do this, I used:

KMP_AFFINITY=logical,granularity=fine ./bin/benchmark

This means the program works as expected (each logical core is bound in ascending order from 0 to 3) in both situations, as the CPU assignments of the bin/my_cpp_program no longer screw with the assignments of bin/benchmark, when one spawns the other. This can be checked as truly occurring by adding verbose to the comma-separated list.

Upvotes: 0

Related Questions