Principles

Up: Contents Next: The Contiguous Routines Previous: Introduction

The design tries to adhere to the following principles:

No extra memory motion, even of single words. A single cache miss can generate enormous delays (on the order of one microsecond in some systesms).
Minimal number of procedure calls.
Efficient support of MPI_Bsend, particularly for short messages.
The common contiguous case should be simple and fast.
The general data structure case should be managed by the ADI, even if it involves simply moving data into and out of contiguous data buffers (putting those buffers under control of the ADI).
The ADI should be usable by itself (without the entire MPI implementation), simplifying testing and tuning of the ADI.

Most of these are clearly desirable. One descision that is not obvious is the support for general datatypes. For reasons of maximum efficiency, particularly for regular but not contiguous datatypes (e.g., strided vectors), it is important that the ADI have the opportunity to handle these types. On the other hand, particularly when porting to a new platform, requiring that the ADI handle the general MPI datatypes imposes a substantial burden on the implementor. The design in this document shows how the general datatypes can be supported using only routines that move contiguous arrays of bytes, and the model implementation contains this support.

This design seems to require more functions (and hence more code) than the previous ADI design but really does not. It simply separates out the individual cases that need to be implemented.

The routines described in this document are expected to be implemented as macros, allowing the use of multiple instances of the routines for implementations that support multiple ADIs. In addition, most of the routines contain an MPI communicator argument that is used as a container for any special information; this may be omitted by particular implementations of the ADI to shorten the argument lists.

One objective that is hard to achieve is to limit the number of redundant memory operations. This particularly impacts error checking; if all of the error checks are made in one place (say, at the top of a routine like MPI_Send), and the same data is used later, possibly in a different routine, the data will be loaded twice. Because the ADI interface is not intended for general use, we place the error checking nearest to the first use of the data. For items like the message tag and buffer, this testing will be in the ADI routines. For items like the datatype and communicator, it may be most natural to test these before calling the ADI routines. For example, the ADI has separate routines for contiguous and noncontiguous data, requiring that the datatype be tested first.

Note that some errors can be detected most efficiently by the ADI routines (for example, in routines that take a datatype and buffer, a null or invalid buffer is detected most easily as the data is being moved, since an MPI datatype may specify an absolute address offset for a buffer, allowing a null buffer as an argument). These routines should also detect message truncation on a receive, and handle any resource limitations. Thus, there is no uniformly convenient place to do all error checking.

In implementing this ADI, there are some operations that will be common to many different implementations. For example, many implementations will need routines to manipulate message queues and the pack and unpack non-contiguous data. The model implementation will provide these.

One final complication comes from the requirement in MPI 1.1 that MPI objects like MPI_COMM_WORLD be compile-time constants. This is much stronger than the MPI 1.0 requirement that they be unchanging between MPI_INIT and MPI_FINALIZE. This essentially requires that the values for these objects be predefined integers, particularly when considering the Fortran versions (which require PARAMETER values). As a result, the types of the user-visiable objects like MPI_Comm, MPI_Datatype, etc. are ints. These ints are translated into pointers to the appropriate structures; the ADI never sees the original int values. Thus, the device interface is given struct MPIR_DATATYPE * instead of MPI_Datatype and struct MPIR_COMMUNICATOR * instead of MPI_Comm. Both of these structures contain a self field that provides the integer value that corresponds to the pointer. It is the responsibility of the code that calls the ADI routines to ensure that the pointers are valid.

The rest of this papers is organized as follows. Section The Contiguous Routines describes the routines for sending and receiving contiguous data. Section Noncontiguous Operations describes the routines for non-contiguous data, and describes how they can be implemented by using the contiguous routines. Section Other Functions describes various service and informational routines. Section Special Considerations describes special considerations and issues in implementing the ADI. Finally, Section Examples presents some example message-passing programs that use only the ADI routines.

Up: Contents Next: The Contiguous Routines Previous: Introduction