CODEWITHSUNDEEP
mpi
Patrick Sanan
Patrick Sanan

Reputation: 2485

When is an additional call to MPI_Win_sync() required when using MPI-3 shared memory operations?

Documentation and examples concerning MPI-3 shared memory operations often include statements like the following, copied from a recent presentation from Intel (my emphasis):

// Start passive RMA epoch
MPI_Win_lock_all (MPI_MODE_NOCHECK, win);

// write into mem array hello_world info
mem[0] = rank;
mem[1] = numtasks;
memcpy(mem+2, name, namelen);
MPI_Win_sync (win);    // memory fence - sync node exchanges
MPI_Barrier (shmcomm); //time barrier

Passive RMA synchronizations are needed for MPI SHM updates. The performance assertion MPI_MODE_NOCHECK hints that the epoch can begin immediately at the target. Note that on some platforms one more MPI_Win_sync would be needed after the MPI_Barrier to ensure memory consistency at the reader side.

Which platforms require this additional call to MPI_Win_sync()? What characteristics of these platforms lead to this requirement? Do I need this additional call on "standard" systems?

Upvotes: 3

Views: 364

Answers (1)

Jeff Hammond
Jeff Hammond

Reputation: 5652

I reviewed this presentation before it was published, along with another person who helped write the MPI-3 RMA chapter, so I know the context for that comment.

The issue here is the memory model of the platform, which usually boils down to the CPU architecture. The x86 memory model is very strong, and total-store ordering (TSO) ensures one doesn’t need memory fences in some scenarios that would otherwise require them on e.g. DEC Alpha. I use Alpha as the example because it has the weakest memory model of any widely used CPU ever.

Anyways, in practice but not in theory (i.e. the MPI standard), a call to MPI_Barrier will synchronize memory, but calling MPI_Win_sync ensures this on x86 at least. The second instance after the barrier would be necessary on a platform where a load fence was required to read data written and store fenced by another thread.

I can draw a picture if it helps.

Upvotes: 0

Related Questions