Cancelling a message is a complex problem. The MPI standard has very strong requirements on the cancel operation, while the ``typical'' user often wishes only to handle cancelling a nonblocking receive that has not started. This case arises in multibuffering algorithms, where more nonblocking receives may be started than will eventually be used. In this case, cancelling the operation is usually quite simple; in the MPICH implementations, it usually means no more than removing the request from the posted receive queue (the queue of receives posted but which have not matched any send).
Cancelling requests in other states is much more complicated. For example, cancelling an MPI_Isend that has started but not completed can be very difficult. Again, we do not wish to add a lot of overhead to a request. A successful cancel of a receive must mark a request as complete, with the status field count in a receive set to MPID_REQUEST_CANCELLED (so that MPI_Test_cancelled can report it). There is also some text in the standard about cancels succeeding, suggesting that cancel can fail.
The MPID design only requires cancels of receives for which no matching message has yet arrived; this was the case that users actually wanted. Other cancel operations may fail. Note that a ``failed'' cancel still returns MPI_SUCCESS (as implied by the MPI standard).
The cancel routines are
MPID_SendCancel( handle, &error_code ) MPID_RecvCancel( handle, &error_code )