Rationale for three modes

In this section we present a performance-based rationale for having three modes (short, eager, and rendezvous) in a message-passing send.

For this analysis, we will assume that the cost of sending a single message of n bytes is s + rn. We further assume that the size of the ``control'' packet (or header) is a fixed h bytes, and that the cost to copy a byte within a process is c. Finally, let f be the fraction of messages that arrive before the matching receive is posted. Then the cost of the three protocols can be summarized as

The various cn terms represent the cost of moving the data within the receiving process. For example, in the short protocol case, the data is received in the control packet and must be copied into the user's buffer. If the message was unexpected, it needs to be copied twice.

With the values here, we can compute the message size where each protocol should be used. For example, the switch between the short and eager protocol should happen when

In other words, when the code of the extra data copy equals the latency to send a separate message, switch to the eager protocol.

We can do the same thing for eager to rendezvous.

Again, when the cost of sending an additional control message (s+rh) is the same as the cost of copying the data (fcn), switch to the rendezvous protocol. We can approximate this in the case that to just

which is just (1/f) times the size for the switch from short to eager.

An important effect not included here is the cost in a rendezvous send of the receiver waiting for the sender to respond to the ``go ahead'' message. The effect of this is to move the point where the rendezvous protocol should be used for performance reasons to larger message sizes. For example, if there might be an average delay d, the formula for switching between eager and rendezvous becomes

if (for example, a timer interrupt every few hundred milliseconds), then n can be very large; larger perhaps than any message that is likely to be sent.

The analysis could be inverted to solve for d instead of n, if the largest allowed n for the eager protocol (because of system buffer limitations, for example) is already known. In this case, if the time d does represent an occasional timer interrupt, then the cost of this interrupt itself (particularly when it discovers nothing to do) needs to be included as well. Such implementation-specific analysis are left to the reader.