Have to Send MPI Message Twice for Some Reason

Question

I've been working on an MPI project in which the slaves all send data back to the master. For some reason the master will only receive the data if I do 2 consecutive Sends in a row. This is very odd and I think it is causing some other weird problems I am getting. Any idea what would cause this? I think the first send is sending some kind of junk data or something. The sends are the exact same line of code though.

EDIT: Code below...

if (numProcs > 0)
    MPI_Barrier( MPI_COMM_WORLD ) ; //only wait if there are other processes to wait for

if (rank != 0)
{
    MPI_Send(handArray, 10, MPI_DOUBLE, 0, TAG_HAND, MPI_COMM_WORLD);
    MPI_Send(handArray, 10, MPI_DOUBLE, 0, TAG_HAND, MPI_COMM_WORLD);
}
//8. After the main loop the master process receives and sums together the hand counts array
//   from each slave process.
else
{
    int activeProcs = numProcs - 1;
    getHandsFromSlaves( activeProcs, handArray );

then the master proceeds to print some data...

Here is the getHands FromSlaves method. Please note I have also tried using blocking calls for this as well with the same problems.

void getHandsFromSlaves( int& activeCount, double handTotals[10] ){

static MPI_Request request;
static int msgBuff, recvFlag;
static double handBuff[10];
MPI_Status status;

while (activeCount > 0)
{
    if( request )
    {
        // Already listening for a message

        // Test to see if message has been received
        MPI_Test( &request, &recvFlag, &status );
        //cout << "TAG: " << status.MPI_TAG << " SOURCE: "<< status.MPI_SOURCE  << " ERROR: " << status.MPI_ERROR << endl;
        if( recvFlag )
        {
            // Message received
            if( status.MPI_TAG == TAG_HAND )
            {
                cout << "Hand Received!" << endl;

                for(int m = 0; m < 10; ++m)
                {
                    handTotals[m] += handBuff[m];
                }

                activeCount--;
            }
            else
            {
                //error report... what happened?
                cout << "TAG: " << status.MPI_TAG << " SOURCE: "<< status.MPI_SOURCE  << " ERROR: " << status.MPI_ERROR << endl;
            }

            // Reset the request handle
            request = 0;
        }
    }

    if( !request && activeCount > 0 )
        // Start listening again
        MPI_Irecv(&handBuff, 10, MPI_DOUBLE, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &request);
}
}

Answer 1

Well, you're probably trying to process one too many messages because your request variable is undefined on entering your getHandsFromSlaves() routine. Since on enter, request is almost certainly non-zero, you immediately try to MPI_Test for a message something even though you haven't posted an Irecv.

In fact, there's a lot of really strange things about the code excerpt posted here. Why are the local variables static? Why would you implement your own busywait on MPI_Test() instead of using MPI_Wait()? Why are you using non-blocking receives at all if you're not doing anything useful between receives? And indeed, if you're just summing up all of the arrays anyway, why are you doing individual point-to-point receives at all instead of doing an MPI_Reduce()?

The following much shorter code seeems to do what you're trying to do above:

#include <stdio.h>
#include <mpi.h>


int main (int argc, char **argv) {

    int rank, numProcs;

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &numProcs);
    double handArray[10];
    double handTotal[10];

    for (int i=0; i<10; i++)
        handArray[i] = rank + i;

    if (rank == 0)  // Since apparently rank 0 doesn't do anything
    {
        for (int i=0; i<10; i++)
            handArray[i] = 0;
    }

    MPI_Reduce(handArray, handTotal, 10, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);

    if (rank == 0) {
        printf("Hand Totals= \n");
        for (int i=0; i<10; i++)
            printf(" %lf ", handTotal[i]);
        printf("\n");
    }

    MPI_Finalize();
}