Main Page   Modules   Data Structures   File List   Data Fields   Globals   Related Pages  

IX. Receiving Messages

Similarly to message sends, message receives in GM are regulated by a simple token-passing mechanism: Before a message can be received, the client software must provide GM a receive token that allows the message to be received and specifies a buffer to hold the received data.

After initialization, the client implicitly possesses all gm_num_receive_tokens() receive tokens. The client software grants receive tokens to GM by calling gm_provide_receive_buffer()(PORT,BUFFER,SIZE,PRIORITY), indicating that GM may receive any message into BUFFER as long as the size and priority fields of the received message exactly match the SIZE and PRIORITY fields passed to gm_provide_receive_buffer(). Eventually, GM will use the buffer indicated by MESSAGE and SIZE to receive a message of the indicated SIZE and PRIORITY. Unlike some messaging systems, GM requires that the SIZE of the received message match the token size exactly. GM will not use the next larger sized receive buffer when a receive buffer of the correct size is not available. All receive buffers passed to gm_provide_receive_buffer() must DMAable. They must also be aligned or be within memory allocated using gm_dma_calloc() or gm_dma_malloc() to ensure that messages can be DMAed into the buffer, and must be at least gm_max_length_for_size()(SIZE) bytes long.

Typical GM clients will provide at least 2 receive buffers for each size and priority of message that might be received to maximize performance by allowing one buffer to be processed and replaced while the network is filling the other. However, 1 receive buffer for each size-priority combination is sufficient for correct operation. Additionally, it is almost always a good idea to provide additional buffers for the smallest sizes, so that many small messages may be received while the host is busy computing. There is no need to provide tokens for receives smaller than gm_min_message_size().

After providing receive tokens, code may poll for pending events using gm_receive_pending()(port), which returns a nonzero value if a receive is pending or zero if no event has been received. gm_next_event_peek()(struct gm_port *P, gm_u16_t *SENDER) can also be used to peek at the event at the head of the queue. The return value is the event type (zero if no event is pending). The SENDER parameter will be filled with the sender of the message if the event is a message receive event. The client may also poll for receives using gm_receive()(PORT), which returns a pointer to a event structure of type gm_event_t. If no recv event is in the receive queue, a pointer to a fake receive event of GM_NO_RECV_EVENT will be returned. The event returned by gm_receive() is only guaranteed to be valid until the next call to gm_receive().

There are several variants of gm_receive() available, all of which can safely be used in the same program.

gm_receive() returns the first pending receive event or GM_NO_RECV_EVENT if none is pending.

gm_blocking_receive() returns the first pending receive, blocking if necessary. This function polls for receives for 1 millisecond before sleeping, so it should generally be used only if the polling thread has a dedicated processor.

gm_blocking_receive_no_spin() returns the first pending receive, blocking if necessary. This function sleeps immediately if no receive is pending. It should be generally used in environments with more than one thread per processor.

Once the client has obtained a receive event from one of these three functions, the client should either process the event if the client recognizes the event, or pass the event to gm_unknown() if the event is unrecognized. All fields in the receive event are in network byte order, and must be converted to host byte order as specified in section X. Endian Conversion.

The client is not required to handle any receive events, and may simply pass all events to gm_unknown(), but any useful GM program will handle GM_RECV_EVENT or GM_HIGH_RECV_EVENT in order to access the received data. The receive event types that the client software may choose to recognize are as follows (GM internal events are not listed):

`event->sent.message_list' points to a null-terminated array of `void' pointers, which are message pointers from earlier gm_send() calls that have completed successfully. For each pointer in this array, a send token is implicitly returned to the client.

These events are just like the normal GM_RECV_EVENT and GM_HIGH_RECV_EVENT events, but indicate that the sender port id is the same as the receiver port id. Most GM programs should handle these events directly just like they handle normal receive events.

These types indicate that a small-message receive occurred with the small message stored in the receive queue for improved small-message performance. The PEER event types indicate that the sender port number is the same as the port number. The HIGH event types indicate that the message was sent with high priority.

If your program uses any small messages that are immediately processed and discarded upon receipt, then your program can improve performance by processing these messages directly. If after examining the message your program determines that it needs the data copied into the buffer, it can either call gm_memorize_message() to do so or can pass the event to gm_unknown().

Note that although the receive data is in the receive queue and no receive buffer was used to store the received message, the client must have provided an appropriate receive buffer before the receive could take place, and this buffer is passed back to the client in the fast receive event. If the client needs to store the data `*message' past the next call to gm_receive(), then the client should copy `*message' into `*buffer' using gm_memorize_message(), which is simply a version of bcopy() optimized for copying aligned messages. After calling gm_memorize_message(), the fast receive event becomes equivalent to a normal receive event.

Although the number of receive events may seem daunting at first glance, almost all of the event types can be ignored. The following receive dispatch loop is fully functional for a nontrivial application that accepts messages ports, accepts only small control messages sent with high priority, and accepts low priority messages of any size:

     {
       struct gm_port *my_port;
       gm_recv_event_t *e;
       void *some_buffer;
       ...
       while (1) {
         e = gm_receive (my_port);
         switch (gm_htohc (e->recv.type))
           {
           case GM_HIGH_RECV_EVENT:
        /* Handle high-priority control messages here in bounded time */
        gm_provide_recv_buffer (my_port,
                                gm_ntohp (e->recv.buffer),
                                gm_ntohc (e->recv.size),
                                GM_HIGH_PRIORITY);
        break;
     
           case GM_RECV_EVENT:
        /* Handle data messages here in bounded time */
        gm_provide_recv_buffer (my_port, some_buffer,
                                gm_ntohc (e->recv.size),
                                GM_LOW_PRIORITY);
        break;
     
           case GM_NO_RECV_EVENT:
        /* Do bounded-time processing here, if desired. */
        break;
     
           default:
        gm_unknown (my_port, e);
           }
       }
     }

However, the following implementation is slightly faster because it handles control messages without copying them into the receive buffer:

     {
       struct gm_port *my_port;
       gm_recv_event_t *e;
       void *some_buffer;
       ...
       while (1) {
         e = gm_receive (my_port);
         switch (gm_ntohc (e->recv.type))
           {
           case GM_FAST_HIGH_PEER_RECV_EVENT:
           case GM_FAST_HIGH_RECV_EVENT:
        /* Handle high-priority control messages here in bounded time */
        gm_provide_recv_buffer (my_port,
                                gm_ntohp (e->recv.buffer),
                                gm_ntohc (e->recv.size),
                                GM_HIGH_PRIORITY);
        break;
     
           case GM_FAST_PEER_RECV_EVENT:
           case GM_FAST_RECV_EVENT:
        gm_memorize_message (gm_ntohp (e->recv.message),
                             gm_ntohp (e->recv.buffer),
                             gm_ntohl (e->recv.length));
           case GM_PEER_RECV_EVENT:
        /* Handle data messages here in bounded time */
        gm_provide_recv_buffer (my_port, some_buffer,
                                gm_ntohc (e->recv.size),
                                GM_LOW_PRIORITY);
        break;
     
           case GM_NO_RECV_EVENT:
        /* Do bounded-time processing here, if desired. */
        break;
     
           default:
        gm_unknown (my_port, e);
           }
       }
     }

Any receive event not recognized by an application must be passed immediately to gm_unknown(), as in the example above. The function gm_unknown() will free any resources associated with the event that the client application would normally be expected to free if it recognized the type. Also, additional, undocumented event types will be received by an application and are handled by gm_unknown(). These messages can be used for supporting features such as GM alarms and blocking receives.

The motivation for putting small messages in the receive queue despite the fact that doing so might require a receive-side copy is the following set of observations:

A large fraction of small receive messages are control messages that can be processed immediately upon reception, and consequently do not need to be copied into the more permanent buffer to survive calls to gm_receive().

The cost of performing an additional DMA to place the message in the buffer, rather than in the receive queue, is actually more expensive for very small messages than having the host perform the copy.

Therefore, placing small received messages in the receive command queue rather than in the more permanent receive buffer enhances performance and is worth the added complexity.

To prevent program deadlock, the client software must ensure that GM is never without a receive token (buffer) for any potentially received message for more than a bounded amount of time. Generally, except for the case of message forwarding described in the next chapter, this means that after each successful call to gm_receive() the client will call gm_provide_receive_buffer() to replace the receive token (buffer) with one of the same SIZE and PRIORITY before the next call to gm_receive() or gm_send(). If such a deadlock condition exists for too long (on the order of a minute) or too often (a significant fraction of a one-minute interval), then remote sends directed at the receiving port will time out.


Generated on Mon Nov 3 15:39:27 2003 for GM by doxygen1.2.15