All message sends are non-blocking and return immediately. This corresponds to MPI's ``immediate'' send mode. Whether packing is enabled or not, messages are non-overtaking as with MPI.
An AUTOPACK send operation is complete when the message has been passed to MPI, and MPI is done using the message buffer. There are some requirements for send completion:
Rationale: If the user has enabled message packing, packages are only sent when full. User allocation of a message send buffer which is too large to fit into the remaining space of a package causes the package to be sent and a new one allocated. A call to AP_flush() forces the library to send any partially full packages. It is good programming practice to call AP_flush() even if packing is not enabled, so that in the future packing may be enabled easily.
Rationale: Parameters maxreq_proc and maxreq (see AP_setparam()) govern the number of simultaneous MPI send requests the library will make. Once either limit has been reached, the library will defer sending any additional messages. The status of prior MPI send requests is checked when the user calls AP_check_sends(). Whenever a send request completes, the library will automatically post as many deferred messages as possible. A negative return value indicates there are no remaining deferred sends; a 0 return value indicates all sends are complete and their buffer space has been freed.
As with MPI, completion of a send operation does not necessarily imply that the destination has received the message.
For convenience, the library will also try to process deferred sends whenever AP_recv() retrieves new messages from MPI. While progress will be made sending deferred messages, the user will not be aware if any remain and must eventually call AP_check_sends().