CODEWITHSUNDEEP
mpistdout
sancho.s ReinstateMonicaCellio
sancho.s ReinstateMonicaCellio

Reputation: 15561

mpirun retaining redirected stdout

I am running in a cluster. I tried to run my executable with 4 different forms:

  1. In serial, with

    myexec

    This starts giving output in stdout right away, as expected.

  2. In serial, redirecting stdout and stderr, with

    myexec > out-err.log 2>&1

    This starts giving output in out-err.log right away, as expected (verified with cat out-err.log in another terminal).

  3. In parallel, with

    mpirun -n 2 myexec

    This starts giving output in stdout right away, as expected.

  4. In parallel, redirecting stdout and stderr, with

    mpirun -n 2 myexec > out-err.log 2>&1

    This retains output until job is finished (due to completion or time allowance).

Is there any way of having stdout/stderr "flushed" at runtime in case 4, so I can check out-err.log?

Upvotes: 6

Views: 2120

Answers (1)

sancho.s ReinstateMonicaCellio
sancho.s ReinstateMonicaCellio

Reputation: 15561

This is a known feature/issue with redirection in mpi. I found how to solve this:

  1. Add export OMPI_MCA_opal_event_include=poll in ~/.bashrc, or

  2. Add opal_event_include=poll in ~/.openmpi/mca-params.conf (create the dir and/or file if they do not exist).

The sources used to get info are:

https://www.cfd-online.com/Forums/openfoam-installation/162664-openfoam-2-4-0-openmpi-epoll-warning-parallel-job.html

https://github.com/open-mpi/ompi/issues/341

https://www.open-mpi.org/doc/v2.0/man1/mpirun.1.php#sect20

Upvotes: 4

Related Questions