CODEWITHSUNDEEP
pthreadsmpi
user2403959
user2403959

Reputation: 33

Calling MPI functions from multiple threads

I want to implement the following thing using MPI and Pthreads but facing some error:

Each processor will have 2 threads. Each processor's one thread will be sending data to other processors and the other thread will be receiving data from other processors. When I am implementing it, it is giving segmentation fault error with some messages like "current bytes -40, total bytes 0, remote id 5".

Just for testing purpose, when I am using only one thread per processor and that is either sending or receiving data, then the errors do NOT occur.

I found the info "In general, there may be problems if multiple threads make MPI calls. The program may fail or behave unexpectedly. If MPI calls must be made from within a thread, they should be made only by one thread." in the following link: https://computing.llnl.gov/tutorials/pthreads/

I want to use two threads per processor where one thread will use MPI_Send function to send some data and the other thread will receive MPI_Recv function to receive data without using any locking mechanism. Does anyone has any idea how to implement this or how to use multiple threads to call MPI functions without using mutex or locking mechanism?

Here is the code:

int rank, size, msg_num;

// thread function for sending messages
void *Send_Func_For_Thread(void *arg)
{
    int send, procnum, x;
    send = rank;

    for(x=0; x < msg_num; x++)
    {
        procnum = rand()%size;
        if(procnum != rank)
            MPI_Send(&send, 1, MPI_INT, procnum, 0, MPI_COMM_WORLD);
    }

// sending special message to other processors with tag = 128 to signal the finishing of sending message

    for (x = 0; x < size; x++)
    {
        if(x != rank)
            MPI_Send(&send, 1, MPI_INT, x, 128, MPI_COMM_WORLD);    
    }

    pthread_exit((void *)NULL);
}


// thread function for receiving messages

void *Recv_Func_For_Thread(void *arg)
{
    MPI_Status status;
    int recv, counter = 0;

    while(counter != size - 1)
    {
        MPI_Recv(&recv, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
        if(status.MPI_TAG == 128)
            counter++;
    }

    pthread_exit((void *)NULL);
}


int main(int argc, char **argv)
{
    void *stat;
    pthread_attr_t attr;
    pthread_t thread[2];

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);   // rank -> rank of this processor
    MPI_Comm_size(MPI_COMM_WORLD, &size);   // size -> total number of processors

    srand((unsigned)time(NULL));

    msg_num = atoi(argv[1]);

    pthread_attr_init(&attr);
    pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);

    // thread 0 will be sending   messages
    pthread_create(&thread[0], &attr, Send_Func_For_Thread, (void *)0);

    // thread 1 will be receiving messages
    pthread_create(&thread[1], &attr, Recv_Func_For_Thread, (void *)1);

    pthread_attr_destroy(&attr);

    pthread_join(thread[0], &stat);
    pthread_join(thread[1], &stat);

    cout << "Finished : Proc " << rank << "\n";

    MPI_Finalize();
    pthread_exit((void *)NULL);
    return 0;   
}

Compile:
========

module load mvapich2/gcc;        mpicxx -lpthread -o demo demo.cpp

Run:
====
mpiexec -comm mpich2-pmi demo 10000000

I ran this program with 3 processors and got segmentation fault.

Upvotes: 2

Views: 3997

Answers (2)

user2403959
user2403959

Reputation: 33

It worked with only one line change with MPICH2.

Instead of using MPI_Init, use the following line:

int provided;
MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);

Thanks all of you for your help and prompt replies!

Upvotes: 1

janneb
janneb

Reputation: 37208

(Since you haven't provided an example, the following is just speculation.)

You must initialize MPI using MPI_Init_thread() instead of MPI_Init(). If I understand your explanation correctly, the "required" argument must have the value MPI_THREAD_MULTIPLE. If MPI_Init_thread() then returns a lower level thread support in the "provided" argument, it means that your MPI implementation doesn't support MPI_THREAD_MULTIPLE; in that case you must do something else. See http://www.mpi-forum.org/docs/mpi-20-html/node165.htm .

Upvotes: 2

Related Questions