what order do I get messages coming to MPI Recv from MPI_ANY_SOURCE,

Question

I am implementing a hub/servers MPI application. Each of the servers can get tied up waiting for some data, then they do an MPI Send to the hub. It is relatively simple for me to have the hub waiting around doing a Recv from ANY_SOURCE. The hub can get busy working with the data. What I'm worried about is skipping data from one of the servers. How likely is this scenario:

• server 1 and 2 do Send's
• hub does Recv and ends up getting data from server 1
• while hub busy, server 1 gets more data, does another Send
• when hub does its next Recv, it gets the more recent server 1 data rather than the older server2

I don't need a guarantee that the order the Send's occur is the order the ANY_SOURCE processes them (though it would be nice), but if I new in practice it will be close to the order they are sent, I may go with the above. However if it is likely I could skip over data from one of the servers, I need to implement something more complicated. Which I think would be this pattern:

• servers each do Send's
• hub does an Irecv for each server
• hub does a Waitany on all server requests
• upon completion of one server request, hub does a Test on all the others
• of all the Irecv's that have completed, hub selects the oldest server data (there is timing tag in the server data)
• hub communicates with the server it just chose, has it start a new Send, hub a new Irecv

This requires more complex code, and my first effort crashed inside the Waitany call in a way that I'm finding difficult to debug. I am using the Python bindings mpi4py - so I have less control over buffers being used.

Answer 1

It is guaranteed by the MPI standard that the messages are received in the order they are sent (non-overtaking messages). See also this answer to a similar question.

However, there is no guarantee of fairness when receiving from ANY_SOURCE and when there are distinct senders. So yes, it is the responsibility of the programmers to design their own fairness system if the application requires it.