More on nonblocking communication

In applications where the time to send data between processes is large, it is often helpful to cause communication and computation to overlap. This can easily be done with MPI's non-blocking routines.

For example, in a 2-D finite difference mesh, moving data needed for the boundaries can be done at the same time as computation on the interior.

MPI_Irecv( ... each ghost edge ... ); 
MPI_Isend( ... data for each ghost edge ... ); 
... compute on interior 
while (still some uncompleted requests) { 
    MPI_Waitany( ... requests ... ) 
    if (request is a receive) 
        ... compute on that edge ... 
    }