Traditional network protocols supported the movement of bits from source

Transcription

1 18 CHAPTER Network Protocols P. M. Melliar-Smith Louise E. Moser Traditional network protocols supported the movement of bits from source to destination. With the inadequate communication media of the past, that was achievement enough. But communication media are improving quite rapidly, though perhaps not rapidly enough, and are becoming cheaper, faster, more flexible, more dependable, and ubiquitous. The network protocols of the future will be designed specifically to support applications, and application requirements will determine the communication services that are provided. Ingenious protocol mechanisms will no longer be required to overcome the inadequacies of the network infrastructure. Communication services are what is important to the users; protocol mechanisms should remain invisible to them. Many network protocols are built hierarchically as a stack of protocols, as shown in Figure The implementation of each protocol in the stack exploits the services provided by the protocols below it. The service provided by each protocol is freestanding and does not depend on whether the implementation of that protocol exploits other protocols. One approach to assembling a protocol stack is to allow the user to construct the stack by choosing protocols from a toolkit of microprotocols [491, 548]. In this chapter, we address the topic of network protocols their characteristics and the services they provide. First, we consider the grid applications discussed in Chapters 3 through 6 and the generic types of communication services they need (Section 18.1). In Section 18.2, we identify four different classes of network protocols that provide these types of services. In Section 18.3, we discuss different types of message delivery services needed by

2 Network Protocols Application Application Protocol service logical protocol physical connection 18.1 Protocol stack. FIGURE the applications. In Section 18.4, we address the issue of maintaining consistency of replicated information, which leads into a discussion in Section 18.5 of group communication and group membership protocols. In Section 18.6, we consider the issues of resource management, including flow control, congestion control, latency, and jitter. With these services and issues in mind, we then present in Section 18.7 an overview of several of the more interesting recently developed protocols. Finally, we discuss in Section 18.8 the future of network protocols APPLICATION REQUIREMENTS The classical network protocol provides a point-to-point, sender-initiated data transfer service, delivering messages in the order in which they were sent, with good reliability and with throughput as the measure of performance. However, each application has its own reasons for communicating and its own types of information to be communicated, which may impose very different requirements on the network protocols. Teleimmersion and collaborative virtual environments (see Chapter 6) are some of the most challenging applications for network protocols because of their varied, demanding requirements. These applications require the following protocols: Data streaming protocols for audio and video, often multicast at high rates

3 18.1 Application Requirements 455 Reliable data transport protocols for collaboration, again often multicast Coordinated control protocols and membership protocols for collaboration Protocols for the integration of complex, independently developed modules Realtime instrumentation applications (see Chapter 4) also have demanding requirements and are currently limited by a reluctance to commit such critical applications to the unreliability of the Internet. These applications require the following protocols: Data transport protocols that provide reliable delivery of messages and also unreliable timestamped delivery of messages containing the most recent available data Coordinated control protocols with fault tolerance Protocols for the integration of complex, independently developed modules Data-intensive applications (see Chapter 5) are still in their infancy. These applications will undoubtedly require the following protocols: Data transport protocols for the efficient, rapid, and reliable transport of huge datasets Protocols for the integration of complex, independently developed modules Protocols for the encapsulation and integration of existing modules and movement of program code to remote sites where the data are located Distributed supercomputing applications (see Chapter 3), both on interconnected supercomputers and on clusters of small computers, are perhaps the best developed of the four types of applications. These applications require the following protocols: Low-latency, reliable data transport protocols, sometimes multicast Low-latency control and synchronization protocols that are scalable to quite large numbers of nodes Protocols for the encapsulation and integration of existing modules and movement of program code to remote sites at which the computation is to be performed

4 Network Protocols 18.2 CLASSES OF NETWORK PROTOCOLS Despite the multitude of network protocols that have been proposed, a few major classes of protocols are starting to emerge. We consider four of these below. There is a common tendency to describe the mechanisms of the protocols, even though it is the protocol services that are important to the users. We endeavor to emphasize the services Data Transport Protocols Data transport protocols are the workhorses of the Internet and are exemplified by the venerable but highly effective Transmission Control Protocol (TCP). They typically provide reliable, source-ordered delivery of messages with excellent flow control and buffer management. Most data transport protocols are point to point, but multicast protocols are becoming more common. The Xpress Transfer Protocol (XTP) [34, 526] is an example of a data transport protocol that has been designed specifically to achieve high performance, particularly for parallel computations. Quite different is the Scalable Reliable Multicast (SRM) protocol [194], a reliable data transfer protocol intended for multicasting to large groups over the Internet, as required, for example, in collaborative virtual environment applications. XTP and SRM are described in more detail in Sections and , respectively Streaming Protocols Streaming protocols are typically used for audio, video, and multimedia and also for certain instrumentation applications. These applications do not require reliable delivery and can accept message loss, particularly if selectiveloss algorithms are used. Multicasting is common in these applications, but unreliable source-ordered delivery suffices. Even when the traffic is bursty, flow control is ineffective, and bandwidth reservation is preferable. Low jitter is important. An example of a streaming protocol is the Real-Time Transport Protocol (RTP) [498], described in Section , which might be used in conjunction with the Resource reservation Protocol (RSVP) [76, 584], described in Section Unfortunately, the datagram orientation of the Internet does not yet support streaming protocols well; the connection-oriented, negotiated quality-of-service approach of ATM generally provides a better foundation for streaming protocols.

5 18.2 Classes of Network Protocols Group Communication Protocols What distinguishes group communication protocols from the data transport and streaming protocols described above is that group communication protocols are concerned with more than just movement of data. In a distributed system, to permit effective cooperation and to provide fault tolerance, a group of several processes must all keep copies of the same data, and the copies of the data must be kept consistent as the application executes. Maintaining consistency is difficult for the application programmer, particularly in the presence of faults. Group communication protocols assist application programmers in maintaining the consistency of replicated data by maintaining the memberships of process groups and by multicasting messages to those process groups. The InterGroup Protocols (IGPs) [56], described in Section , are group communication protocols being developed for distributed collaborative virtual environments and realtime instrumentation applications, to allow scientists and engineers to collaborate and conduct experiments over the Internet. The IGP protocols might also be suitable for other applications. They pay special attention to scaling to large sizes and to minimizing the effect on latency of long propagation times across a large network Distributed Object Protocols A new area of protocol development is distributed object protocols for heterogeneous distributed systems, exemplified by the Internet Inter-Orb Protocol (IIOP) [388] developed by the OMG for CORBA (see also Chapter 9). CORBA supports the use of existing legacy code, provides interoperability across diverse platforms, and hides the distributed nature of the computation and the location of objects from the application. However, the overheads of CORBA s remote method invocation are currently still high. IIOP, described in Section , allows ORBs from different vendors to work together. CORBA and IIOP are well suited for distributed collaborative virtual environments and realtime instrumentation applications, which involve the integration of complex distributed systems from existing software. Also important is Java s Remote Method Invocation (RMI) [290], described in Section , which supports the movement of code to remote machines (see also Chapter 9). While similar in some respects to CORBA, Java RMI provides less support for the integration of complex systems and no support for existing code. Currently, Java is less efficient than CORBA, but improved performance should come soon. Java and RMI are particularly well suited for data-intensive and distributed supercomputing applications, as these can

6 Network Protocols require movement of programs. The integration of Java with CORBA achieves the best of both technologies TYPES OF MESSAGE DELIVERY SERVICES The most basic form of message delivery service is unreliable message delivery, which provides only a best-effort service with no guarantees of delivery. Unreliable message delivery is used for audio and video streaming protocols, such as are needed in teleimmersion. Most applications require a more stringent service, reliable message delivery, which ensures that each message sent is eventually received, possibly after multiple retransmissions, despite loss of messages by the communication medium. Many recent network protocols extend the point-to-point service of the traditional network protocol to a multicast service [151]. Multicasting allows efficient delivery of the same information to several, possibly many, destinations a service needed particularly by collaborative virtual environments but also by other applications. Multicasting is conceptually simple but involves many complications in the protocol mechanisms to achieve efficiency. Typical multicast protocols provide only one-to-many multicasting, where a single sender sends to multiple destinations, but collaborative virtual environments require many-to-many multicasting. Another important criterion for a message delivery service is the order in which messages are delivered, including causally ordered delivery, sourceordered delivery, group-ordered delivery, and totally ordered delivery, as described below. Causally ordered delivery [328] requires that (1) if process P sends message M before it sends message M and process Q delivers both messages, then Q delivers M before it delivers M, and (2) if process P receives message M before it sends message M and process Q delivers both messages, then Q delivers M before it delivers M. Delivery of messages in causal order prevents anomalies in the processing of data contained in the messages, but it does not maintain the consistency of replicated data. Source-ordered delivery, or FIFO delivery, requires that messages from a particular source are delivered in the order in which they were sent. For multimedia streaming protocols, such as are required for teleimmersion, sourceordered delivery suffices. Data distribution, such as in data-intensive applications, also requires only source-ordered delivery. Group-ordered delivery requires that, if processes P and Q are members of a process group G, then P and Q deliver the messages originated by the processes in G in the same total order. A reliable group-ordered message

7 18.4 Maintaining Consistency 459 Messages multicast Messages delivered A 1 A 2 A 3 B 1 B 1 B 2 B 3 B 2 C 1 C 2 C 3 B 1 C 2 A 1 C 1 B 2 A 3 C 3 A 1 A 2 C 1 B 3 C 2 C 3 C 1 B 2 A 2 B 3 Unreliable source-ordered delivery A 1 A 2 A 3 B 1 A 2 B 1 B 2 B 3 B 1 B 2 C 1 C 2 C 3 B 1 C 2 A 1 C 1 B 2 B 3 C 2 A 3 C 3 A 1 A 2 C 1 B 3 A 3 C 2 C 3 C 1 A 1 B 2 C 3 A 2 A 3 B 3 Reliable source-ordered delivery A 1 A 2 A 3 B 1 A 2 B 1 B 2 B 3 B 1 A 2 C 1 C 2 C 3 B 1 A 2 A 1 C 1 B 2 B 3 C 2 A 3 C 3 A 1 C 1 B 2 B 3 C 2 A 3 C 3 A 1 C 1 B 2 B 3 C 2 A 3 C 3 Reliable group-ordered delivery 18.2 FIGURE Types of message delivery services, where A, B, and C are processors and A 1, A 2, and A 3 are messages sent by processor A, and so forth. delivery service helps to maintain the consistency of replicated data when processes that constitute a group have copies of the same data and must update that data. Totally ordered delivery requires that, if process P delivers message M before it delivers message M and process Q delivers both messages, then Q delivers M before it delivers M.Totally ordered delivery is important where systemwide consistency across many groups is required, as in realtime instrumentation applications. Various types of message delivery services are illustrated in Figure MAINTAINING CONSISTENCY In a distributed system, processing and data may be replicated for increased reliability or availability, or faster access. In particular, several processes may perform different tasks of the application and may each have a copy of the

8 Network Protocols Client A 1 A 2 A 3 Replicated data at server Client A 1 B 1 A 2 B 2 B 3 C 1 C 2 A 3 C 3 B 1 B 2 B 3 Replicated data at server Client A 1 B 1 A 2 B 2 B 3 C 1 C 2 A 3 C 3 C 1 C 2 C FIGURE Total ordering on messages helps to maintain the consistency of replicated data. same data so that they can cooperate. Alternatively, several processes may be replicated for fault tolerance, with each replica performing the same task of the application and holding a copy of the same data. In both cases, the copies of the replicated data must be kept consistent as the application executes. Group communication protocols [56, 62, 162, 372, 401, 548] help to maintain the consistency of replicated data. Such protocols distribute update messages to all processes holding copies of the replicated data using a multicast service. They provide a reliable, totally ordered message delivery service, which ensures that each process performs the same sequence of updates in the same order, as shown in Figure This simplifies the application programming required to maintain the consistency of the replicated data. Because they are designed to support fault tolerance in addition to coordination of distributed computation, these protocols are very robust and handle a wide range of faults, including processor faults and network partitioning. They include excellent fault detection, group membership, and recovery protocols that provide a consistent view of faults and membership changes systemwide [62, 400]. Early group communication protocols were inefficient, but performance has improved. Current group communication protocols operate very effi-

9 18.5 Group Membership 461 ciently in the benign private world of a LAN, as efficiently as other protocols despite the more elaborate services they provide. Effective strategies are, however, still being developed for the hostile world of the Internet [56], where resources must be shared with unknown and uncooperative users. Applications for these protocols can be purely computational (e.g., distributed supercomputing applications), where tasks might be allocated to processors by a distributed scheduler, which is coordinated by a group communication protocol. Alternatively, these protocols can be used to mediate human interaction, as in a collaborative virtual environment, where a group communication protocol can ensure that all users see the same updates to a shared workspace. The protocols can also be used in the control or monitoring of a physical device, as in a realtime instrumentation application, where different processes control different parts of the instrumentation, coordinated by a group communication protocol GROUP MEMBERSHIP A process group [62] is a set of processes that collaborate on an application task. Messages related to the application task are multicast to the members of the process group. A distributed application may consist of hundreds of process groups, and a process group may contain hundreds of processes. The processes may join and leave the groups voluntarily, as in a collaborative virtual environment, or because a process or processor has failed, as in a realtime instrumentation application. Naming mechanisms that work well for simple applications on small, stable networks are inappropriate for large complex dynamic applications. For example, if a processor fails, its processes may be replaced by other processes on different processors. Thus, messages are not addressed to a specific process on a specific processor. Rather, each message is addressed to the process group and is delivered to all of the processes that have joined that group. This strategy must be supported by mechanisms that allocate unique names to process groups and by a name server. Some applications and many protocol mechanisms need to know the current group membership for each process group so that, for example, a distributed scheduling algorithm can assign tasks on a distributed supercomputer. It is also important to establish a precise relationship between message ordering and membership changes. For example, if state is being transferred to a new process on another processor, it is essential to establish whether the messages were processed before the membership change and thus are reflected

10 Network Protocols Q R S Membership is Q, R, and S S fails P Membership is Q and R Q R Continued operation before fault is detected P joins group Membership is P, Q, and R State transfer P Q R Messages processed after state transfer Messages processed before state transfer 18.4 FIGURE When state is transferred from one process to another, it is important that both processes agree on which messages should be processed before the state transfer and which should be processed after. in the state being transferred, or whether the messages were processed after the membership change and thus should be processed by the new process, as shown in Figure The concept of ordering membership changes relative to data messages is called virtual synchrony [62]. The maintenance of group membership is quite difficult indeed theoretically impossible without the use of a fault detector but several effective practical membership protocols have been developed. As shown in Figure 18.5, the typical approach involves the following phase: A discovery phase of broadcast solicitations and responses that establishes which processes are alive and can communicate

11 18.5 Group Membership 463 Q R S Membership is Q, R, and S Normal operation S fails Membership algorithm operating Fault detected Discovery phase Commitment phase Recovery phase Membership algorithm done P P joins group Last few messages of old membership are delivered Membership is P, Q, and R Normal operation resumes with new messages 18.5 The phases of a membership protocol. FIGURE A commitment phase using a form of two-phase commit that determines the membership A recovery phase in which messages from the prior membership are collected and delivered Liveness is ensured by repeating these steps as often as is required to complete the algorithm while at each step excluding at least one additional process thought to be faulty in the previous membership attempt. This guarantees termination in finite time but at the expense of possibly allowing, in unfavorable circumstances, each nonfaulty process to form a membership containing only itself. In practice, with appropriate values of the time-outs, the approach is robust and effective. The problem with such a membership protocol, and many other protocols that maintain network topology, connectivity, or distance information, is that the cost is proportional to N 2, where N is the number of processes in the group. Furthermore, the interval between membership changes is inversely proportional to N. AsN becomes large, the system may spend too much of

12 Network Protocols its time in the membership protocol. Worse still, to ensure virtual synchrony, many group communication systems suspend message delivery during membership changes. As N becomes large, they may deliver no messages at all. Research is under way into weaker forms of membership that scale better while remaining useful to the applications, such as the collaborative virtual environments and distributed supercomputing applications, that may involve many processes RESOURCE MANAGEMENT The great advantage that the Internet has demonstrated over traditional telecommunication networks is increased sharing of network resources, which leads to reduced costs. Many network connections have intermittent and bursty traffic. Where a network resource can be shared between enough connections, typically more than 20, quite high utilization of the resource can be obtained without serious degradation of the service provided to each connection. Multiple bursty traffic sources will, however, always be able to overwhelm the network; thus, flow control and congestion control will remain important Flow Control and Congestion Control With modern communication equipment, packet or cell loss is almost always caused by congestion, either in the switches and routers of the network or in the input buffers at the destinations, rather than by corruption of packets or cells by the medium. Consequently, the TCP protocol uses packet loss as an indication that it should reduce its transmission rate. The TCP backoff algorithm, which employs rapid rate reduction on packet loss and slow rate increase in the absence of packet loss, is very effective for data transmission, as for example in data-intensive and distributed supercomputing applications. It allows many TCP connections to share the network fairly and efficiently without explicit coordination. The strategy suffers, however, from several problems. First, it is not appropriate for data streaming, such as the multimedia connections of a teleimmersion application, where the transmission rate is relatively high but predictable and constant. Such connections are best served by an explicit reservation strategy, such as ATM provides and RSVP has proposed for the Internet. A second problem is that connections that observe the TCP backoff strategy can be squeezed out by rogue connections that do not reduce their transmission rate in the presence of congestion and packet loss. This problem is

13 18.6 Resource Management 465 Messages originated Messages delivered Latency 18.6 FIGURE Latency is the time from message origination at the source to message delivery at the destination(s). Jitter is the variance in the latency. addressed by ATM s traffic-policing mechanisms and, for the Internet, by random early drop (RED) algorithms in Internet switches [193]. RED algorithms explicitly discriminate, by dropping packets, against connections that do not reduce their transmission rate in the presence of congestion. TCP s backoff strategy also becomes less effective for connections that communicate at high rates over long distances. The time to detect packet loss on a 1 Gb/s transcontinental link corresponds to about 5 MB of transmitted data, given the 40 ms round-trip time. At, say, 40 Gb/s, this becomes over 200 MB, and switch buffering requirements become unreasonable Throughput, Latency, and Jitter Critical performance metrics for network protocols and for the applications are throughput, latency, and jitter. Throughput is the number of user data bits communicated per second. Latency is the time from message origination at the source to message delivery at the destination(s), and jitter is the variance in the latency, as shown in Figure Because of the inadequate performance of the network infrastructure, protocols have focused on maximizing the throughput of the network, which has been the bottleneck. As network throughput has improved, attention has shifted to latency and jitter. Protocol mechanisms that increase the throughput of the network may also increase the latency. Long latencies are undesirable for some applications, such as distributed supercomputing and realtime instrumentation and control. Protocols have been developed, particularly for distributed supercomputing, with mechanisms to reduce the latency caused by the protocol stack and the operating

14 Network Protocols Three sources A, B, and C multicasting A 1 A 2 A 3 Input buffer of a receiver B 1 B 2 B 3 A 1 B 1 A 2 C 1 B 2 A 3 C 2 C 3 C 1 C 2 C 3 Buffer overflows; messages are lost B FIGURE Multicast protocols incur a high risk of packet loss at the input buffers of the destinations. system (see Chapter 20). Such mechanisms are effective, however, only for lightly loaded systems in which queuing delays are not significant. For heavily loaded complex systems, high throughput reduces queuing delays and is the most effective approach for reducing overall system latency. Jitter is highly undesirable for audio, video, and multimedia applications, such as teleimmersion. Special protocols that timestamp every packet, such as RTP [498], allow accurate measurement of latency, which is effective at helping to reduce or mask jitter. Also important for multimedia applications, particularly those that use MPEG encoding of streams, are protocols and switches that select intelligently the packets to be dropped when congestion occurs [193] Multicast/Acknowledgment Mechanisms Good performance of group communication protocols, and even the feasibility of Internet video broadcasting, requires that multicasting be able to reach many destinations with a single transmission, sharing the bandwidth and avoiding wasteful multiple transmissions of the same information. With a modern high-performance network and multiple sources multicasting simultaneously, as in a collaborative virtual environment, it is easy to transmit information faster than the receivers can process it, resulting in a bottleneck at the input buffers of the receivers, as shown in Figure When using highperformance LANs, rather than the slower Internet, the best group communication protocols focus their flow control mechanisms on preventing input buffer overflow [401]. Many recent protocols exploit negative acknowledgments rather than positive acknowledgments to achieve higher performance and higher reliability [194]. Positive acknowledgment protocols, also called sender-based protocols,

15 18.6 Resource Management 467 Acks Nack Multicast message 18.8 FIGURE Positive and negative acknowledgments. With many receivers in a multicast group, there is an implosion of acknowledgments at the sender. require the receiver to send an acknowledgment to the sender if it has received the message. If the sender does not receive an acknowledgment within a certain period of time, it retransmits the message. In contrast, negative acknowledgment protocols, or receiver-based protocols, require the receiver to send a negative acknowledgment to the sender if it has not received the message within a certain period of time. With a large number of receivers in a multicast group, positive acknowledgment protocols can result in the sender being overwhelmed by an implosion of positive acknowledgments, as shown in Figure 18.8, making a negative acknowledgment protocol more appropriate. To achieve reliable delivery, positive acknowledgments are typically combined with negative acknowledgments. This combination allows messages to be removed from the sender s buffers when those messages are no longer needed for possible future retransmission. As larger amounts of data are transmitted over longer distances at higher speeds, the buffers must become larger, and buffer management becomes more important Translucency The limited performance of current communication networks, coupled with rapidly growing and changing demands on network protocols, has encouraged

16 Network Protocols translucency, or access, by the application programmer to the mechanisms of the network protocols. Examples are the following: Network-aware applications adapt their own behavior and demands on the network to reflect the current network load, latency, or rate of message loss. This strategy may allow applications to operate more effectively under adverse network conditions. The disadvantage is that the strategy increases application programming costs. Building a better network may be cheaper. Application-level framing allows the application to be aware of, and even manipulate directly, the actual packets or frames transmitted over the communication medium. The advantage can be lower overheads for applications such as teleimmersion; the disadvantage is that the application program loses portability and adaptability to future networks. Microprotocol toolkits allow the user to assemble the protocol stack by choosing the protocols from a toolkit of microprotocols [491, 548]. The advantage is that a skilled user can construct a protocol suited to the needs of the application. In general, however, network protocols constructed by the user from a microprotocol toolkit are less efficient than more highly integrated protocol implementations because the interfaces between the microprotocols incur overhead and inhibit code optimization. Additional inefficiency is caused by microprotocols that must be designed to be freestanding so that they can be used with or without other microprotocols. As high-performance networks become readily available, as the networking environment stabilizes, and as the needs of applications become better understood, the advantages of translucency will diminish, and the costs of custom network programming in the application will be harder to justify, even for performance-sensitive applications such as teleimmersion and distributed supercomputing. The predominant consideration will be the convenience and efficiency of a standard, fully integrated protocol providing a simple, welldefined, and stable service to the application Achieving Adequate Network Performance The primary requirement of adequate network performance is adequate bandwidth, which is currently a problem but will become less of a problem in the future. Efficient and cost-effective use of the network will, however, depend on sharing the cost of the bandwidth with many users, and on a pricing mechanism that restrains the growth in demand and also funds the increase

17 18.7 Example Network Protocols 469 in available bandwidth. Other resources, such as buffer space, must also be shared. Users who require exclusive reservation of resources, such as ATM AAL1 users, are likely to pay much higher costs than users who agree to share resources. The rapid growth and effectiveness of the Internet depend in part on the low costs made possible by packet-switching protocols that share the available bandwidth among multiple packets simultaneously. The congestion of the Internet results from the lack of an effective pricing mechanism. As networks continue to develop, the greatest contributions of network protocols will result from improved services rather than improved mechanisms. Substantial performance improvements result from transmitting information once only instead of many times in a collaborative virtual environment, or of transmitting the right information in a data-intensive application, or of not needing to transmit the information at all in a distributed supercomputing application. Caching, now used quite extensively to speed access to popular sites on the Web, is an example of a mechanism that improves network performance by reducing the need to transmit information. As yet, other network protocols have made little use of caching beyond buffering messages for possible retransmission after message loss. Improved protocols and improved network performance, resulting from transmitting the right information at the right time, will result from understanding better the services that the applications need EXAMPLE NETWORK PROTOCOLS We describe here a few of the more interesting, recently developed network protocols. The different characteristics and mechanisms of these protocols make them appropriate for different aspects of grid applications such as those considered in this book Xpress Transfer Protocol The Xpress Transfer Protocol (XTP) is a transport-level protocol designed for distributed parallel applications operating over clusters of computers. For flexibility and efficiency, it supports several different communication paradigms, as discussed below. XTP supports both unicasting from a single sender to a single receiver and one-to-many multicasting from a single sender to a group of receivers. Data flows in one direction from the sender to the receiver(s), with control traffic

18 Network Protocols flowing in the opposite direction. Multiple instances, or contexts, of XTP can be active at a node, providing partial support for many-to-many multicasting. Normally, the receivers do not inform the sender of missing packets until the sender asks for such information; however, an option is available that allows the receivers to notify the sender immediately after detecting a packet loss. A control packet indicating lost packets contains the highest consecutive sequence number of any of the packets received by a receiver and the range of sequence numbers of missing packets. Retransmissions are multicast to the group and are either selective or go-back-n. The sender may disable error control but may still ask the receivers for flow control and rate control information. XTP supports both flow control and rate control algorithms, which provide the requested quality of service. Flow control, in which further data is sent when acknowledgments are received for prior transmissions, is appropriate for both data and control. Rate control, in which data is transmitted at a constant rate, is appropriate for data streaming. Flow control considers endto-end buffer space; rate control considers processor speed and congestion. Multicast groups are managed by the user, based on information that the sender maintains about the receivers. The user specifies how the initial group of receivers is formed, and the criterion for admission and removal once the multicast group is established. The sender learns which receivers are active by periodically soliciting control packets. The scalability of XTP is limited by its method of error recovery. Like other sender-initiated error recovery approaches, the sender may be unable to maintain state for large receiver groups, and the throughput of the sender may be slowed by control packet implosion and processing and by retransmissions. Multicasting retransmissions to the entire multicast group uses unnecessary bandwidth on links to receivers that have already received the packet Scalable Reliable Multicast Protocol The Scalable Reliable Multicast (SRM) protocol is a lightweight protocol that supports the notion of application-level framing and is intended for applications such as collaborative virtual environments. SRM aims to provide reliable delivery of packets over the Internet, in that all packets should eventually be delivered to all members of the multicast group; no delivery order, either source ordered or group ordered, is enforced [194]. To receive packets sent to the group, a receiver sends a join message on the local subnet announcing that it is interested in receiving the packets. Each receiver voluntary joins and

19 18.7 Example Network Protocols 471 leaves the group without affecting the transmission of packets to other group members. Each group member is individually responsible for its own reception of packets by detecting packet loss and requesting retransmission. Packet loss is detected by finding a gap in the sequence numbers of the packets from a particular source. To ensure that the last packet in a session is received, each member periodically multicasts a session message that reports the highest sequence number of packets that it has received from the current senders. The session messages are also used to determine the current participants in the session and to estimate the distance between nodes. When a node detects a missing packet, it schedules a repair request (retransmission request) for a random time in the future. This random delay is determined by the distance between the node and the sender of the packet. When its repair request timer for the missing packet goes off, the node multicasts a repair request for the packet, doubles its request timer delay, and waits for the repair. If the node receives a request for the packet from another node before its request timer goes off, it does an exponential backoff and resets its request timer. When a node receives a repair request and it has the requested packet, the node schedules a repair for a random time in the future. This random delay is determined by the distance between the node and the node requesting the repair. When its repair timer for the packet goes off, the node multicasts the repair. If the node receives a repair for the packet from another node before its repair timer goes off, it cancels its repair timer. The request/repair timer algorithm introduces some additional delay before retransmission to reduce implosion and to reduce the number of duplicates. It combines the systematic setting of the timer as a function of distance (between the given node and the source of the lost packet or of the repair request) and randomization (for the nodes that are at an equal distance from the source of the missing packet or repair request). Because the repair timer delay depends on the distance to the node requesting the repair, a nearby node is likely to time out first and retransmit the packet The InterGroup Protocols The InterGroup Protocols (IGPs) are intended for large-scale networks, like the Internet, with highly variable delays in packet reception and with relatively high packet loss rates, and for large-scale applications with relatively few senders and many receivers [56]. The IGPs are particularly appropriate for collaborative virtual environments and realtime instrumentation applications.

20 Network Protocols Each Data packet has in its header a unique source identifier, a sequence number, and a timestamp obtained from the sender s local clock. The sequence numbers are used to detect missing packets from the sender and to provide reliable source-ordered delivery of packets. The timestamps provide reliable group-ordered delivery of packets, while maintaining the causal order of messages sent within the group. Each multicast group consists of two subgroups: the sender group and the receiver group. Superimposed on the underlying multicast backbone (MBone of the Internet) is a forest of trees that includes all of the receivers in the multicast group. This forest of trees is used to collect acknowledgments, negative acknowledgments, and fault detection packets to reduce congestion and to achieve scalability. Data packets and retransmissions are multicast by using the MBone. Only the senders multicast Data packets, but the senders are also receivers in that they receive Data packets. Both senders and receivers may send retransmissions of Data packets and are required to send Alive packets. An Alive packet serves as a heartbeat and also as a positive acknowledgment, since it contains the highest timestamp up to which the node multicasting the Alive packet has received all packets. Each node sends an Alive packet on an infrequent periodic basis, but only to its parent in the tree, and the information in the packet is then propagated up the tree. Upon receiving the information, a root node (i.e., sender) multicasts its Alive packet to the group. If a node detects that it has missed a packet, it sends a Nack (negative acknowledgment) packet. Under high packet loss rates, Nack packets are sent more frequently than Alive packets, but again a node sends a Nack packet only to its parent in the tree. The InterGroup Protocols use approximately synchronized clocks at the senders and receivers, by running a clock synchronization algorithm occasionally or using a Global Positioning System. At minimum, these clocks are Lamport clocks that respect causality [328]. If every member of the sender group is sending packets, the maximum delivery time of a packet is D + τ, where D is the diameter of the group and τ is the interpacket generation time of the slowest sender. The maximum delivery time is kept low by using Alive packets from the senders. To maintain consistency of packet delivery, both senders and receivers must know the membership of the sender group. For scalability, only the senders are responsible for detecting faults of the senders in the sender group and for repairs of the sender group. Before a node can deliver a packet from a particular sender, it must have received a packet, from each sender in the sender group, with a timestamp at least as great as that of the packet to be

21 18.7 Example Network Protocols 473 delivered. Otherwise, the node may deliver a packet from one sender with a higher timestamp before it receives a packet from another sender with a lower timestamp. Both senders and receivers must know the membership of the receiver group, either implicitly or explicitly. This information is used for garbage collection so that a node can remove a Data packet from its buffers when it knows that all nonfaulty members of the receiver group have received the packet and thus that it will never need to retransmit the packet subsequently. Again, for scalability, to maintain the membership of the receiver group, each node is responsible only for fault detection of its children in the tree Real-Time Transport Protocol The Real-Time Transport Protocol (RTP) [498] provides end-to-end delivery services for data with realtime characteristics, such as interactive audio and video (see also Chapter 19). RTP is intended primarily for multimedia conferencing with multiple participants, as in teleimmersion applications, but it is also applicable to interactive distributed simulation, storage of continuous data, and so on. RTP typically runs on top of the User Datagram Protocol (UDP) to utilize its multiplexing and checksum services, and supports data transfer to multiple destinations using multicasting provided by the underlying network. RTP works in conjunction with the Real-Time Transport Control Protocol (RTCP). The Real-Time Transport Protocol provides realtime transmission of data packets, while the Real-Time Transport Control Protocol monitors quality of service and conveys session control information to the participants in a session. Among the services provided by RTP are payload type identification, marker identification, sequence numbering, timestamping, and delivery monitoring. The payload type identifies the format of RTP payload (e.g., H.261 for video), and the marker identifies significant events for the payload (e.g., beginning of a talk spurt). The sequence numbers allow the receivers to detect packet loss and to restore the sender s packet sequence, but may also be used to determine the position of a packet in a sequence of packets as, for example, in video transmission. The timestamp, obtained from the sender s local clock at the instant at which a data packet is generated, is used for synchronization and jitter calculations. RTP itself does not provide any mechanisms to ensure timely delivery or other quality-of-service guarantees. It does not guarantee delivery or prevent out-of-order delivery, nor does it assume that the underlying network provides such services. For jitter-sensitive applications, a receiver can buffer packets

22 Network Protocols and then exploit the sequence numbers and timestamps to deliver a delayed but almost jitter-free message sequence. RTCP is based on periodic transmission of control packets to all of the participants in a session. The traffic is monitored and statistics are gathered on the number of packets lost, highest sequence number received, jitter, and so forth. These statistics, transmitted in control packets, are used as feedback for diagnosing problems in the network, controlling congestion, handling packet errors, and improving timely delivery. In addition, RTCP conveys minimal session control information, without membership control or parameter negotiation, as the participants in a session enter or leave the session. Although RTCP collects and distributes control and quality-of-service information to the participants in the session, enforcement is left to the application Resource reservation Protocol The Resource reservation Protocol (RSVP) [76, 584] is designed for an integrated services Internet and is suitable for applications such as teleimmersion. More details on RSVP are provided in Chapter 19. RSVP makes resource reservations for both unicast and multicast applications, adapting dynamically to changing group memberships as well as to changing routes. RSVP operates on top of the Internet Protocol IPv4 or IPv6 and occupies the place of a transport protocol in the protocol stack. However, it does not transport application data nor does it route application data; rather, it is a control protocol. RSVP is based on the concept of a session, which is composed of at least one data stream or flow and is defined in relation to a destination. A flow is any subset of the packets in a session that are sent by a particular source to a particular destination or group of destinations. RSVP is used to request resource reservations in each node along the path of a flow and to request specific qualities of service from the network for that flow. RSVP is receiver oriented in that the receiver of a flow initiates and maintains the resource reservation used for that flow. Thus, it aims to accommodate large groups, dynamic group memberships, and heterogeneous receiver requirements. It carries a resource reservation request to all of the switches, routers, and hosts along the reverse data path to the source. Since the membership of a large multicast group and the topology of the corresponding multicast tree are likely to change with time, RSVP sends periodic refresh messages to maintain the state in the routers and hosts along the reserved paths. This is referred to as soft state because it is built and destroyed incrementally.

23 18.7 Example Network Protocols 475 Quality of service is implemented for a particular flow by a mechanism called traffic control that includes a packet classifier and a packet scheduler. The packet classifier determines the QoS for each packet, while the packet scheduler achieves the promised QoS for each outgoing interface. During reservation setup, a QoS request is passed to two local decision modules, admission control and policy control. Admission control determines whether the node has sufficient available resources to supply the requested QoS; policy control determines whether the user had administrative permission to make the reservation CORBA GIOP/IIOP CORBA is based on the client-server model and supports interoperability at the user level (language transparency) via the OMG s Interface Definition Language (IDL) and at the communication level via inter-orb protocols [388]. CORBA is also discussed in Chapters 9 and 10. When a client object invokes a method of a server object, the call is bound toastatic stub generated from the IDL specification of the server object, or the operation is invoked dynamically. The stub passes the call and its parameters to the Object Request Broker (ORB), which determines the location of the server object, marshals (encodes and packages) the call and its parameters into a message, and sends the message across the network to the processor hosting the server object. The ORB at the server unmarshals the message and passes the call and its parameters to a skeleton, which invokes the operation and returns the results to the client object, again via the ORB. The CORBA 2.0 specification provides the technology that enables ORBs from different vendors to communicate with each other, known as the ORB Interoperability Architecture or, more specifically, the General InterORB Protocol (GIOP). GIOP is designed to map ORB requests and responses to any connection-oriented medium. The protocol consists of the Common Data Representation, the GIOP message formats, and the GIOP message transport assumptions. The Common Data Representation (CDR) is a transfer syntax that maps every IDL-defined data type into a low-level representation that allows ORBs on diverse platforms to communicate with each other. CDR handles variations in byte ordering (little-endian vs. big-endian) across the different machines hosting the ORBs. It also handles memory alignments to natural boundaries within GIOP messages.