End System Multicast Protocol for Collaborative Virtual Environments

Transcription

1 Mojtaba Hosseini Nicolas D. Georganas End System Multicast Protocol for Collaborative Virtual Environments DISCOVER Laboratory School of Information Technology and Engineering University of Ottawa 800 King Edward Avenue Room Ottawa, Canada Abstract IP Multicasting has been a crucial requirement of many scalable networked virtual environments by providing an efficient network mechanism through which a sender can transmit its information to a large number of receivers without having to send multiple copies of the same data over a physical link. The widespread deployment of IP Multicast has been slow due to some yet unresolved issues, prompting recent efforts in the development of multicasting protocols at the application layer instead of at the network layer. Most of these protocols address the case of a single source streaming media to a large number of receivers in applications such as video-ondemand or live broadcast. Collaborative and distributed virtual environments exhibit different characteristics that in turn necessitate a different set of requirements for application layer multicast protocols. This paper presents an introduction to application layer multicasting as it relates to distributed and collaborative virtual environments and the development of our own end system multicast protocol for multisender virtual teleconference applications. 1 Introduction Presence, Vol. 13, No. 3, June 2004, by the Massachusetts Institute of Technology The scalability of Distributed Virtual Environments (DVE) and Collaborative Virtual Environments (CVE) with regards to the number of simultaneous users they can support has been a topic of much research. In chapter 7 of their book, Singhal and Zyda (1999) address the scalability issue on several levels that can be categorized as belonging to the application layer (e.g., area of interest management, dead reckoning), transport layer (e.g., packet aggregation and compression), network layer and, in particular, multicasting. Multicasting refers to a mechanism by which a source can transmit to only those receivers that have subscribed to a particular multicast group. Multicasting forms an important criterion for the scalability of applications such as media streaming and CVE teleconferencing, as well as large-scale DVEs. However, the Internet was originally designed based on unicasting (one-to-one) principles and could not provide for a source to transmit to a large number of users without causing great inefficiency, spurring the development of multicast capability at the network IP layer or, in other words, IP Multicast (Deering & Cheriton, 1990). Exploiting IP Multicast has provided developers of distributed virtual environments with an easy and efficient way of achieving scalability for their applications (Macedonia, Zyda, Pratt, Brutzman, & Barham, 1995), Hosseini and Georganas 263

2 264 PRESENCE: VOLUME 13, NUMBER 3 as evidenced by the plethora of systems making use of IP Multicast, such as Macedonia, Pratt, and Zyda (1994), Barrus, Waters, and Anderson (1996), and Greenhalgh, Benford, and Reynard (1999), to name a few. IP Multicast is especially effective when coupled with various area-of-interest management schemes where entities are restricted to receive only from those that lie within their sphere of interest or awareness (for instance associating a multicast address with a region or locale in the virtual world.) See Singhal and Zyda (1999) and Greenhalgh et al. (1999) for more details. Though distributed CVE applications typically may not have the scalability requirement to support thousands of users (as is the case for some DVE applications), they can still benefit from the existence of a multicast-capable network, especially if high bit-rate media (e.g., audio and video) are incorporated into the CVE application (e.g., Greenhalgh et al., 1999; Greenhalgh, Purbrick, & Snowdon, 2000; Hosseini & Georganas, 2003). Though it forms a crucial requirement of many applications, such as video-on-demand, live broadcasts, teleconferencing, and collaborative multimedia, as well as some distributed CVE and DVEs, the widespread deployment of IP Multicast has been slow due to yet unresolved issues regarding administration, management, addressing, reliability, quality of service, and congestion control (Diot, Levine, Lyles, Kassem, & Balensiefen, 2000). Though there exist overlay networks connecting IP Multicast islands (the most widespread of which is the MBone; Eriksson, 1994), they involve static setting up of connections or tunnels and are still plagued by the same issues related to IP Multicast. As a result, many Internet Service Providers (ISPs) do not provide multicast capability to home Internet users, leaving most applications to still rely on unicast only. In light of the slow deployment of IP Multicast on the Internet, and encouraged by the success of peer-topeer file transfer applications, recent research endeavors have studied application layer multicast and investigated its cost and benefits relative to IP Multicast (Chu, Rao, Seshan, & Zhang, 2002). Application layer multicast refers to implementing multicast capability at the application layer instead of the network layer, or in other words, constructing a multicast-capable overlay network over a unicast-only infrastructure. Many of the current approaches to application layer multicasting focus on single-source applications such as video-on-demand and live broadcasts. Section 3 will give a brief overview of existing approaches. To the best of our knowledge, there are no attempts at providing application layer multicasting appropriate for DVE or CVE applications. These have characteristics that are significantly different from those of single-source streaming, video-ondemand, and file-transfer applications and therefore have a different set of requirements from the protocols that will support them. In this paper we provide an introduction to application layer multicast as it relates to DVE and CVE applications and present our end system multicast protocol designed for a special class of CVE applications: multisender teleconferencing. 2 Related Work The most widespread attempt at providing multicast capability is through the Multicast Backbone or MBone (Eriksson, 1994), an overlay internetwork connecting IP Multicast-capable networks called multicast islands with unicast connections called tunnels. Expansion of the MBone presents difficulty due to the static setting up of the tunnels, as well as the administration and management issues inherent to IP Multicast. As a result, very few ISPs provide a connection to the MBone for home Internet users. An extension of the MBone but more appropriate for DVE applications, DIVEBone (Frecon, Greenhalgh, Stenius, 1999), is also an application-level overlay network that interconnects IP Multicast islands together and connects MBone unaware applications using proxy servers. DIVEBone suffers from the same shortcomings as MBone regarding deployment and static setting up of proxy servers, as well as administration and management. Facing the lack of availability of IP Multicast to many users, and wishing to accommodate them, many DVEs have opted for the traditional client-server architecture with multiple servers each being responsible for a portion of the workload (e.g., RING: Funkhouser, 1995;

3 Hosseini and Georganas 265 NetEffect: Das, Singh, Mitchell, Kumar, & McGee, MASSIVE-3: Greenhalgh et al., 2000). To these systems, the server fulfills the job of a multicast group by receiving data from users and distributing them to other users directly or via other servers. By creating a multi-unicast point however, the server can quickly become a bottleneck, especially if the DVE involves exchange of high bit-rate media. Other systems (e.g., MASSIVE-1: Greenhalgh & Benford, 1995) allow for the establishment of peer-to-peer connections between users for exchange of their data, but in this way shift the bottleneck to users that have to transmit to several recipients. The recent Application Layer Multicast (ALM) approach couples the concept of peer-to-peer connection between entities with the additional concept of entities forwarding a received stream to other receivers, thereby alleviating the necessity for the original source of the stream to send to all its receivers. The next section presents a brief overview of existing ALM protocols, followed by a discussion of their shortcomings with respect to multisender applications, such as teleconferencing CVEs as well as large scale DVEs, and goes on to present our ALM protocol that, to the best of our knowledge, is one of the first of its kind addressing multisender CVE teleconferencing applications. 3 Application Layer Multicast Overview Recent efforts that investigate the possibility, cost, and benefit of implementing overlay networks at the application layer, as opposed to at the network layer, address the issue at two different levels: at the infrastructure level, and at the host or end system level. Infrastructure-level or proxy-based ALM involves servers or proxies strategically placed on the Internet and connected to one another with unicast connections to form an overlay network (Chawathe, McCanne, & Brewer, 2000; Ratnasamy, Handley, Karp, & Shenker, 2001; Zhuang, Zhao, Joseph, Katz, & Kubiatowicz, 2001; Zhang, Jamin, & Zhang, 2002; Banerjee, Kommareddy, Kar, Bhattacharjee, & Khuller, 2003; Zhang & Hu, 2003; Shi & Turner, 2002). Each host or end system connects to its nearest server/proxy and subscribes to a Figure 1. IP Multicast (left), infrastructure-level ALM (center) and end system level ALM (right). multicast group by informing that server/proxy of the group it wishes to subscribe to. A multicast group is formed by constructing a tree that spans those servers/ proxies that have an interested host connected to them. Subsequently, a host wishing to multicast sends its data to the server/proxy it is connected to, where the data are copied and transmitted to all other servers/proxies on the overlay tree constructed for that particular multicast group. Receiving servers/proxies forward the information to any host part of the session, as well as other servers/proxies on the overlay tree (Figure 1b). In contrast, end system ALM involves the hosts themselves in sharing the responsibility for forwarding information to others by organizing them into an overlay tree (Figure 1c) (Francis, 1999; Mathy, Canonico, & Hutchison, 2001; Pendarakis, Shi, Verma, & Waldvogel, 2001; Banerjee, Bhattacharjee, & Kommareddy, 2002; Chu et al., 2002; Kwon & Fahmy, 2002; Padmanabhan, Wang, Chou, & Sripanidkulchai, 2002; Tran, Hua, & Do, 2003). Figure 1 demonstrates the difference between IP Multicast, infrastructure-level ALM, and end-systemlevel ALM. While IP Multicast relies on network components such as routers to copy and forward data, infrastructure-level ALM does so using servers or proxies connected together via unicast, and end-system-level ALM uses the end systems themselves for forwarding. As can be seen, IP Multicast avoids sending multiple copies of the same data on a physical link but requires routers to keep state information. ALM results in multiple copies of the same data on a physical link but can be constructed and deployed over the existing unicast infrastructure by not requiring changes to current routers on the Internet.

4 266 PRESENCE: VOLUME 13, NUMBER 3 End-system-level ALM (henceforth called End System Multicast, ESM) has the advantage of being flexible, being adaptable to specific application domains, and not requiring any support from the Internet infrastructure, but it puts strain on the hosts by requiring them to forward, may not scale well for large group sizes, and may not be as efficient as infrastructure-level ALM. Infrastructure-level ALM can take advantage of IP Multicast islands, where available, and therefore increase its efficiency relative to ESM, and it also relieves the hosts of any forwarding responsibility with regards to multicast data, but it requires the deployment of dedicated servers and proxies in the infrastructure. Our work focuses on designing an ESM protocol for multisender CVE application. This is in line with our goal of developing CVE frameworks and applications that are immediately deployable over the Internet, as opposed to requiring IP Multicast or dedicated servers or proxies. Existing ESM protocols, however, target single-source applications with a large number of receivers, typically the case in video-on-demand and live broadcast applications. NICE (Banerjee et al., 2002) and ZIGZAG (Tran et al., 2003) create a hierarchical structure of nodes rooted at a single source, while CoopNet (Padmanabhan et al., 2002) constructs multiple trees, each carrying a different subdescription of a single stream, in order to increase robustness. All three target single source streaming to a large number of receivers (thousands or tens of thousands). TAG (Kwon & Fahmy, 2002) constructs an overlay tree rooted at a single source but exploits information about the underlying topology to reduce the delay between the source and its nodes, as well as to avoid duplicate packets on physical links. It uses bandwidth as the secondary metric for making trees and again, it is only appropriate for single-sender applications. CAN-multicast (Ratnasamy et al., 2001) uses delay as the primary metric and ignores the fan-out bandwidth limitation of each node, and is therefore more suitable for file-transfer or largescale event distribution applications. Yoid (Francis, 1999) and TBCP (Mathy et al., 2001) construct a single tree per session and, as will be shown in section 6 of our paper, suffer large rejection rates as a result. ALMI (Pendarakis et al., 2001) targets conferencing applications of smaller scale and uses a centralized mechanism to construct a minimum spanning tree rooted at a single source, but does not regard bandwidth as a primary metric. Narada (Chu et al., 2002) first constructs a mesh of nodes and then makes a spanning tree rooted at a source on top of this mesh. The mesh increases robustness in case of node failure and decreases management in case a tree with a different root is to be constructed. Of these, only Narada and ALMI target teleconferencing applications, but both construct a single tree for a session and assume a single source transmits at any point in time. The characteristics of CVE and DVE applications are different however, requiring ESM protocols to be designed accordingly. 4 DVE Requirements for an ESM Protocol It has already been argued that the current IP Multicast architecture and protocols do not efficiently support large-scale applications such as DVEs (Levine, Crowcroft, Diot, Garcia-Luna-Aceves, & Kurose, 2000). In their paper, Levine et al. cite the requirements for large-scale applications as including: partitioning of receiver sets by interest in content need for fast joins and leaves of multicast groups need for a large number of concurrent multicast groups They go on to highlight the shortcomings of IP Multicast with regards to these requirements. Studying the current ALM architectures and protocols, one can observe that the same shortcomings are carried over from IP Multicast to ALM, leading to the conclusion that there is a need to design ALM protocols appropriate for DVE and distributed CVE applications. However, DVE and distributed CVE encompass a very large and highly variant set of applications, ranging from large-scale military simulations, social virtual environments, teleconferences, and medical and industrial remote training, to multiplayer games, distance education, and remote collaborative engineering design. Each class of application in turn has a widely different set of requirements concerning interactivity, reliability, scalability (both in terms

5 Hosseini and Georganas 267 of content and number of users), synchronization, and consistency. Therefore, it is difficult to conceive of a single generic ALM architecture and protocol that satisfies the requirements of all the different DVE and distributed CVE applications. Instead, we believe that different protocols will coexist, each targeting a different class of applications. In fact, it is one of the advantages of implementing multicasting at the application layer, since it allows different multicast protocols to exploit and be tailored to application-specific requirements without requiring the underlying network to support the resulting plethora of protocols. We have thus far focused our efforts on multisender teleconferencing applications e.g., (Greenhalgh & Benford, 1995; Reynard, Benford, & Greenhalgh, 1998; Baker, Bhatti, Tanguay, Sobel, Gelb et al., 2003; Hosseini & Georganas, 2003; Vertegaal, Weevers, Sohn, & Cheung, 2003). In addition to the characteristics mentioned above, we add the following, more specific to multisender CVE teleconferences: Multiple simultaneous sources Each user is the source of its own (usually high bandwidth) data. This has significant implications for designing ESM protocols since the fan-out bandwidth of each node is limited and may already be used by its own outgoing data, nullifying some of the assumptions of existing protocols (such as the availability of fan-out bandwidth of all receivers for forwarding as well as constructing a single tree for a multiparty session). Number of users and awareness management Though a large-scale teleconference may potentially include thousands of users, there are usually interest/awareness management schemes in place that limit the mutual interaction between a subset of all users. For applications that involve a single user transmitting to thousands of users, single-source ESM protocols such as those already mentioned would be more appropriate. We instead concentrate on teleconferences with a small number of users (4 10) where each user is itself a source in addition to being a receiver. Examples of this have already been cited, and the proof-of-concept application is presented in section 7. Of the four classes of applications that drive multicast deployment as presented in Diot et al. (2000), these correspond to the few-to-few conferencing applications. Soft join/leave events Existing protocols view a leave event as the result of a node experiencing failure or leaving the session and a join event as the result of a new node joining the session. Both of these primitives imply tearing down and construction of links or edges in the overlay graph, which is the case for video-on-demand and live broadcast applications. However, in the context of multisender teleconferences, in addition to these types of leave and join events, there is also the notion of a node leaving a source (e.g., as a result of loss of interest/awareness in a source) and yet remaining a part of the session, allowing it to continue forwarding to its children for a time in order to prevent disruption in the flow of packets until they can be attached to another node. Frequent soft join/leave In addition to fast join/ leaves of multicast groups, there are usually frequent join/leaves as a result of frequent changes in awareness/interest of users. As mentioned already, current efforts to provide multicast at the application layer do not target DVE or distributed CVE applications with the mentioned requirements. The next section presents the design of an ESM protocol appropriate for multisender teleconferencing applications. 5 An ESM Protocol for Multisender CVE Teleconference Applications The core design of any ESM protocol consists of (1) the choice between a distributed or centralized scheme for constructing trees or hierarchies; (2) the choice between a mesh-first approach or a tree-first approach to constructing such trees; and (3) the algorithm and mechanism for constructing, improving, and maintaining trees when users join/leave a tree rooted at a source. What follows is our design of an ESM protocol

6 268 PRESENCE: VOLUME 13, NUMBER 3 and the choices therein as they relate to the characteristics and requirements of multisender videoconferences. 5.1 Centralized vs. Distributed An ESM protocol must choose between a centralized approach and a distributed approach to constructing, improving, and maintaining trees or hierarchies representing the multicast group. While a distributed approach provides better scalability and fault tolerance by distributing the decision making process between many nodes, it is slower than a centralized approach in reacting to changes (e.g., nodes joining or leaving the hierarchy) since a joining node must iteratively search for a node to connect to. It is this iterative search that causes continuous packet loss for seconds and sometimes tens of seconds when, for instance, an intermediate node leaves (Banerjee et al., 2002). As mentioned in the previous section, many DVE and distributed CVE applications employ interest/awareness management to limit the number of users wishing to receive from a source. Furthermore, given the lesser scalability requirement of the applications we are targeting and the frequently changing characteristic of multisender teleconferences (as compared to video-on-demand and live broadcast), we opt for a centralized approach similar to ALMI (Pendarakis et al., 2001) for constructing and maintaining overlay structures. We assume the existence of a Rendezvous Point (RP) whose identity is known to all participants prior to the start of the session. Similar to other protocols, the RP serves as a bootstrap mechanism whereby participants contact in order to join the session but, additionally, in our protocol the RP also contains the centralized decision making component that keeps track of the overlay tree for all sources. Note that the RP may physically reside on one of the client machines. Clients inform the RP of their wish to join/ leave a particular source and, based on the algorithm that will be discussed in section 5.5, the RP decides how to modify the overlay trees in order to accommodate every source sending to every one of its receivers. (Note that it is the decision making process that is centralized and not the stream distribution for each source, which is done in a peer-to-peer manner.) 5.2 Mesh-First vs. Tree-First Existing ESM protocols either construct trees rooted at a source directly from the current list of nodes (e.g., NICE) or construct a richer connected graph called a mesh and then construct overlay trees based on the graph (e.g., Narada). A tree-first approach is more appropriate for single-source applications, while a meshfirst approach allows for robustness and easier management and construction of multiple overlay trees corresponding to multiple sources and is therefore more appropriate for multisender applications. Having decided on a mesh-first centralized approach to constructing, improving, and maintaining overlay trees for the multisender applications, we now present the rest of the protocol for doing so. 5.3 New Node Join Participants join a teleconferencing session by contacting the RP, whose address is known through a mechanism outside of this protocol (e.g., URL is obtained from a website). The RP holds the address of every participant currently part of the session as well as their maximum fan-out bandwidth (obtained either through measurement or explicitly defined by each participant). The RP informs each joining participant of the address of those already in the session. The joining participant then makes a connection to all existing participants and vice versa, thus making a bidirectional Complete Virtual Graph (CVG) between the nodes. This CVG serves as the mesh on top of which trees will be constructed for each source. Note that the RP can physically reside on a client machine. At this point, once again we make a distinction between join and leave events as defined by previous ESM protocols and what we earlier termed soft join and soft leave events in the previous section. Whereas the former refers to nodes joining and leaving the session (e.g., a new node joining or an existing node experiencing failure) and implies construction and tearing down of connections between nodes, the latter refers to an existing node in the session changing the set of media sources it wishes to receive from (e.g., node i requesting to receive video from node j

7 Hosseini and Georganas 269 and node k instead of node m ) and does not necessarily imply tearing down or construction of links, but the resumption or discontinuation of stream transmission between existing peers. 5.4 Node Leave A node gracefully leaving a session informs the RP, which in turn deals with repairing trees that get disconnected as a result of a node leaving. Periodic messages exchanged between a node and the RP are used to detect abrupt failure of a node. Our centralized approach to mesh/tree maintenance simplifies the detection and repair of disconnected trees, which involves identifying nodes that are detached as a result of their parent leaving and reattaching them to the appropriate tree. 5.5 Node Soft Join Once a node is a part of the CVG mesh (as a result of joining the session), it can request to receive data from a set of sources in the session (this would normally be a subset of all sources if interest/awareness management is used). The idea is to find a set of trees rooted at each source and spanning the subset of users it needs to send to, given a maximum out-degree bound for each node. In formal terms, the Multiple-Source Degree- Constrained Spanning (MSDCS) problem is: Given an undirected complete graph G (V,E) with each node V having a maximum out-degree outd_max( ) and given a tuple {s,{t l,...,t M }} for every source s V and its targets t V, find a set of trees rooted at each source and spanning that source s targets such that outd( ) outd_max( ) for all V. If there is a cost (such as delay) associated with each link e E, then the problem may require the spanning trees to also have minimum cost. The problem of finding minimum cost degreeconstrained multicast trees or degree-constrained Steiner trees has been a topic of much research and has been shown to be NP-Complete (Bauer & Varma, 1995; Kortsarz & Peleg, 1998; Ravi, Marathe, Ravi, Rosenkrantz, & Hunt, 2001; Malouch, Liu, Rubenstein, & Sahu, 2002; Konemann & Ravi, 2003). These, however, focus on constructing a single tree and do not consider multiple trees over the same graph. Though there has been some research with regards to constructing multiple trees on a shared graph (Chen, Gunluk, & Yener, 2000), they still only provide a bound on the worst case (maximum) degree of any node as opposed to guarantees on the individual maximum degree for every node as is required for our protocol. In the context of infrastructure-level application layer multicast, Shi and Turner (2002) provide several heuristic routing algorithms for finding a single minimum-diameter, degree-constrained, balanced spanning tree in a complete graph such that the probability of accommodating future multicast trees is increased. In section 6 we compare our own heuristic algorithm for the MSDCS problem with those presented by Shi and Turner. Note also that traditional multicast routing algorithms such as DVMRP (Deering & Cheriton, 1990) and PIM (Deering, Estrin, Farinacci, Jacobson, Liu, et al., 1994) do not take into account the out-degree limitations of nodes and are only concerned with constructing a single tree. We have therefore devised our own heuristic algorithms for the stated problem. The details of the algorithm are given in the appendix. Here we draw attention to some of the main points. The global formulation of the tree finding problem presented above (MSDCS) considers finding spanning trees for a set of sources. In practice however, multicast sessions are created dynamically as nodes join and leave trees rooted at different sources. Rearranging all existing trees every time a node joins or leaves can cause instability. Therefore, instead of addressing the global formulation of the problem, it is worth investigating the possibility of attaching a joining node to the requested tree without significantly altering the existing trees (called the Dynamic Steiner Tree problem in the case of Steiner trees: Aharoni & Cohen, 1998). We have thus devised two algorithms: one that addresses the dynamic joining and leaving of nodes, called Softjoin, and another, called SoftjoinAll, that addresses the global formulation of the problem and is used when Softjoin does not reach a solution (see appendix for details). Softjoin begins by searching the existing tree rooted

8 270 PRESENCE: VOLUME 13, NUMBER 3 at the requested source, s, for a node, p, that can parent the requesting node, t, while respecting its maximum out-degree and possibly keeping the cost between s and t at a minimum. If such a node is not found, we investigate the relocation of a spare out-degree from a node not on the tree of s to a node on the tree of s. The relocation algorithm rearranges the tree of other sources in order to transfer a spare out-degree to the tree of source s so that the requesting node t can be attached. If the Softjoin algorithm fails to find a solution SoftjoinAll investigates the rearrangement of all trees in order to accommodate the request (i.e., the global formulation of the problem). A feature of the SoftjoinAll algorithm is that it accounts for nodes that are not on the list of those to be spanned by a particular tree but yet can be used as intermediaries for forwarding when necessary. In the context of videoconferencing for instance, such nodes are called reflectors or Multipoint Control Units (MCUs). Commercial videoconferencing tools use reflectors or MCUs to serve as multi-unicast points to allow a user to transmit to a number of users (CUWorld, 2003). In our algorithm, the concept of a reflector or MCU is generalized to any node that, despite not wanting to receive a stream itself, can help in forwarding streams to others. In this way, in addition to considering a permanent reflector, a user can temporarily become a reflector when its fan-out is idle but it is not a recipient of a source that cannot otherwise accommodate all its receivers (see SoftjoinAll in appendix for details). Section 5.7 illustrates the two algorithms through an example. 5.6 Node Soft Leave We take a simplistic approach to a soft leave event (i.e., a user requests to stop receiving from a source and yet remains a part of the conferencing session) in the interest of causing less disruption of stream delivery. If the requesting node has more than one child in the tree of the source, we ignore the soft leave request and use the requesting node as a reflector (a node that does not itself wish to receive a stream and yet is used to forward to more than one node). If, however, the node making the soft leave request has one or no children in the tree Figure 2. An example of degree-constraint routing. of the source, it is simply removed from the tree and the disconnected child reconnected to the parent of the leaving node. 5.7 An Example We now illustrate some of the characteristics of the routing algorithm whose details can be found in the appendix with a simple example. Consider a complete graph with nodes 0,1,2,3 having maximum out-degrees of (1,1,2,3) streams respectively as shown in Figure 2(a) (maximum out-degrees are displayed at the side of each node). Figure 2(b) shows a possible configuration of trees at a given point in time. Consider at this point a request from user0 to receive the stream of user1. Traditional routing algorithms that only consider a single tree do not find a solution since the source (node1) and its children (node2) do not have any spare out-degree to accommodate a new node. Our algorithm however relocates the spare degree of node3 to node2 and can therefore attach node0 to node2 as shown in Figure 2(c). Now consider the tree configuration shown in Figure 2(d). User2 s wish to receive the stream of user0 can only be accommodated if user3 is used as a reflector : that is, if it does not itself wish to receive a stream but can be used to forward a stream to more than one user because of its spare out-degree as shown in Figure 2(e).

9 Hosseini and Georganas 271 At this point we illustrate a situation where it is impossible to find a solution (i.e., to find a set of trees spanning each source s targets and respecting every node s out-degree limit) even though the total number of receivers is not more than the total out-degree available from all users. Given the configuration shown in Figure 2(f), user2 s request to receive from user0 can not be accommodated even though the number of requests ( ) is the same as the total out-degree available ( ). 5.8 Not Finding Paths There are three main categories of cases where our routing algorithm does not find a solution: (1) when the total out-degree required for the session (i.e., all join requests) exceeds the total of the maximum outdegree of all nodes and therefore there is no possible solution; (2) when the total out-degree required for the session is less than or equal to the total of the maximum out-degree but it is still impossible to accommodate all requests (see example in the previous section); (3) when it is possible to find a solution and yet the algorithm does not find it. The first case of failure can be addressed as an admission control mechanism at the application level by restricting the total number of withstanding soft join requests to be no greater than the total maximum outdegrees of all users (see section 7 for an example of a proof-of-concept application that achieves such a restriction). We are currently investigating the cases where our algorithm does not find a solution to discern what percentage of cases it is actually impossible to find a solution for. Note that there are theoretically feasible solutions that may not be trivial to implement, an issue alluded to in Hosseini and Georganas (2003). 6 Initial Evaluation The primary metrics for evaluating different ALM protocols are the average and worst-case stress and stretch of the overlay tree or hierarchy (Chu et al., 2002). Stress refers to the number of identical packets on the same physical link, while stretch refers to the number of physical hops between a source and receiver. A shortest path first tree constructed on top of an IP Multicast-capable network represents the near-optimal case by avoiding multiple packets on the same link and minimizing delay between the source and its receivers. ALM protocols however must strike a balance between low-stretch, high-stress trees and low-stress, highstretch trees. Dealing with multiple trees on the same graph with degree-constrained nodes however first brings up the question of feasibility before the question of optimality. In other words, given nodes with low node out-degrees, how often is it possible to construct trees for each source, each spanning that source s receivers? The question need not be answered for singlesource applications because the receivers of that source can be assumed to be available for forwarding a stream to others. In multisender applications, however, each receiver can itself be a source of its own data and cannot be automatically assumed to be available for forwarding. For the construction of multiple multicast trees over the same network of degree-constrained nodes, the rejection rate is a metric of how successful a routing algorithm is in accommodating requests to join a multicast tree (Shi & Turner, 2002). It is towards examining the feasibility question that our first evaluation of the ESM protocol is directed. 6.1 Evaluation Methodology Our evaluation involves testing the routing algorithm under different configurations and observing the percentage of rejected requests to join a multicast tree, as a result of the algorithm not finding a solution. We compare the rejection rate of our algorithm with two other overlay multicast protocols described by Shi and Turner (2002) called Compact Tree (CT) and Balanced Degree Allocation (BDA). CT represents a greedy algorithm for finding a path from a source to a receiver and is commonly used in single-source application layer multicast protocols that use fan-out bandwidth as a metric (e.g., Yoid: Francis, 1999; TBCP: Mathy et al., 2001; and OMNI: Banerjee et al., 2003). In CT, a joining node is attached to the requested tree as close (in

10 272 PRESENCE: VOLUME 13, NUMBER 3 terms of distance or delay) to the source as possible as soon as a spare degree is available, resulting in lowstretch trees with high degree nodes at the top of the tree. BDA, on the other hand, constructs more balanced trees in order to allow for the construction of future trees on the same overlay structure while still respecting the degree bound of each node. BDA has the best (lowest) rejection rate of all algorithms presented by Shi and Turner (2002), in accommodating multiple multicast trees. BDA is similar to the routing algorithms of CANmulticast (Ratnasamy et al., 2001) and ZIGZAG (Tran et al., 2003) in distributing the forwarding load, but with the key difference that BDA respects the heterogeneous out-degree of all nodes. We therefore compare the rejection rate of our algorithm with the best (BDA) and most commonly used (CT) algorithms. For the purpose of comparison, simulations were carried out with [4 9] nodes, each with a fan-out maximum of between [1 5], where nodes randomly join and leave trees rooted at other nodes. (This is equivalent to teleconference participants randomly changing the sources they wish to receive from.) To see the effect of different configurations on the rejection rate, we examine all possibilities of fan-out maximums that can be assigned to nodes (a total of M N possibilities for N users and a range of out-degrees between 1 and M). The configurations represent the different set of permutations of maximum out-degrees and consequently the heterogeneity of fan-out bandwidth available to each user. Testing all possible configurations, as opposed to randomly assigning out-degree nodes as is commonly done, gives more confidence as to the generality of the results (but it is only feasible to do such a test for a small number of nodes). The generation of random join/leave events is restricted such that a node does not request to join a tree it is already attached to and does not request to leave a tree it is not attached to. Also, the total number of withstanding join events is limited to no more than the total of maximum out-degrees of all nodes. For example, for the out-degree configuration of (1,1,2,3), the total number of withstanding join requests can be no larger than 7, thus avoiding the cases where accommodating all requests is obviously impossible since the Figure 3. Rejection rate in the case of 4 nodes. number of join requests is greater than the total number of available out-degrees. For each possible configuration, random join/leave events are generated and the number of requests that are rejected by all three algorithms recorded. 6.2 Simulation Results Figure 3 shows the percentage of rejected requests for our algorithm for all possible configurations involving 4 nodes, plotted against the total maximum outdegree of all nodes and the number of nodes with a maximum out-degree of only 1. The worst-case rejection rate of 9% occurs when all nodes have a maximum out-degree of only 1 (the 1,1,1,1 configuration). As the number of nodes with out-degree of 1 decreases, so does the rejection rate. Also, as the total out-degree available increases, the rejection rate decreases. In the context of teleconferences, these results emphasize that the existence of participants whose out-degree bandwidth can only accommodate sending one stream (e.g., a user with a dial-up modem connection in an audio/ video teleconference) reduces the feasibility of constructing overlay trees. This is more the case as the number of such nodes/users increases.

11 Hosseini and Georganas 273 however (4 10), the low 90-percentile rejection rate (less than 10%) is an encouraging sign of the feasibility of using ESM for multisender teleconferencing applications. 6.3 Algorithm Complexity Figure Percentile rejection rate versus number of users. Although all three algorithms (CT, BDA, and our own) have O (N 3 ) run-time complexity (where N is the number of nodes), our algorithm is more complex to implement since it more rigorously attempts to rearrange trees to accommodate the greatest number of requests, once a trivial solution is not found. Having illustrated the effect of different out-degree configurations in a heterogeneous environment on the rejection rate, we wish to examine the performance of our algorithm compared to other algorithms in terms of rejection rate or, in other words, the ability of each to find solutions. Figure 4 shows the 90-percentile rejection rate of our algorithm compared to CT and BDA algorithms discussed earlier and presented by Shi and Turner (2002). As mentioned already, CT represents a commonly used minimum spanning tree algorithm in overlay multicasts, while BDA represents the algorithm with the best (lowest) rejection rate presented by Shi and Turner (2002). As can be seen, the 90-percentile rejection rate is significantly improved by our algorithm compared to both CT and BDA. Figure 5 shows the comparison of average and maximum rejection rates for the three algorithms. In the context of teleconferencing applications, the higher rejection rate in the case of a larger number of nodes emphasizes that with an increase in the number of bandwidth-constraint participants, it becomes increasingly more difficult to address the multicasting issue at an end system level. Instead, more cooperation from the infrastructure is needed to make teleconferences with a larger number of users possible or the system must, alternatively, reduce the fidelity of the media types involved (such as by reducing the quality of video in a videoconferencing application in order to increase the out-degree available). For a small number of users 7 Proof-of-Concept 3D Videoconferencing Application In order to demonstrate the viability of using ESM in conjunction with multiparty videoconferencing applications, we have implemented a proof-of-concept 3D videoconferencing application that can make use of the ESM protocol described in previous sections. The application is similar to previous efforts trying to provide mutual awareness and spatial consistency (Reynard et al. 1998), gaze awareness (Taylor & Rowe, 2000; Baker et al., 2003; Vertegaal, Weevers, Sohn, Cheung, 2003) or awareness driven quality of service (Greenhalgh et al., 1999) in multiparty videoconferencing applications. Implementation details of our application are given elsewhere (Hosseini & Georganas, 2003), including how the occlusion-based awareness management mechanism limits the number of streams the system must deal with, as well as how the application is designed to better utilize an end-system multicast protocol. The basic idea is to limit the number of simultaneous streams requested by all participants such that participants with spare fan-out bandwidth can reduce the load of another participant by forwarding its stream. In such applications, participants request to join and leave the streaming session of other participants depending on their awareness (Figure 6). If such an application is deployed over our ESM protocol, we are interested in

12 274 PRESENCE: VOLUME 13, NUMBER 3 Figure 5. Average (left) and maximum (right) rejection rate. Figure 6. Awareness-driven videoconferencing in 3D environment (user4 receives video only from 1 and 3 since 2 is out of view). Figure 7. Success rate for different configurations in 4-person videoconferences. knowing how often we can expect the protocol to find application layer multicast trees for every source. For this purpose we replayed the six (4 6 minute) teleconferencing sessions reported in Hosseini and Georganas (2003) with different configurations of maximum outdegrees assigned to the 4 participants and observed the success rate, as well as the number of changes that occurred when accommodating a request. Figure 7 shows the percentage of time a request was accommodated by a single change to the topology, by multiple changes, and when no solution was found (the request was rejected) for the different configurations of out-degree maximums assigned. There were a total of 419 join requests (and 385 leave requests) during the six teleconference sessions. Note that we have grouped the different configurations of out-degrees between 1 5 according to how many users exist with outdegree of 1. For example the configurations (3,3,1,1) as well as (4,2,1,1) etc. are grouped together in Figure 7 as those having two nodes with out-degree of 2. As the number of users with maximum out-degree of 1 increases, the probability that no solution is found increases from 0% to 4% (for all the permutations of 5,1,1,1 configurations). Furthermore, the probability that a single change in topology can accommodate a request decreases from 96% to 69% as the configuration includes more users with out-degree of 1. Overall, the high success rate of our routing algorithm, especially in the case of peers with out-degree maximums of more than one, is encouraging. More extensive tests are required as a part of our future work in order to evaluate the cost of the ESM protocol in terms of delay and link stress. At this point however, our initial results suggest

13 Hosseini and Georganas 275 the potential of our ESM protocol in the context of multisender videoconferences, and the potential of an awareness-driven 3D videoconferencing application to be able to take advantage of such a protocol in order to accommodate 4 10 users and be deployable over the Internet without a need for IP Multicast support from the infrastructure. 8 Related Issues and Future Work The key parameter and metric for constructing trees in our ESM protocol is the fan-out bandwidth of each user. Fan-out or upload bandwidth available in a link connecting a user to the Internet is of a dynamic nature and can fluctuate during a teleconference session depending on the amount of congestion on the interconnecting links (the upper bound on upload bandwidth of course depends on physical characteristics of the connecting link). It is therefore important and necessary for our future work to include a mechanism whereby the ESM protocol can appropriately react to congestion. The incorporation of an integrated congestion management scheme such as Balakrishnan, Rahul, and Seshan, (1999) would additionally provide for efficient multiplexing of multiple flows, while ensuring proper and stable congestion behavior and fairness to TCP flows running in parallel. As mentioned in section 5.5, finding a minimum cost tree with degree bounds on each node is an NP- Complete problem. Therefore the heuristic routing algorithm developed mainly attempts to construct multiple trees on top of the same graph while respecting each node s out-degree bound without providing any guarantee on the delay. We therefore plan to investigate the possibility of providing (soft) bounds on the delay for each tree while still respecting the out-degree constraints. It is also of interest to investigate the incorporation of other Quality of Service parameters (priority as well as different quality streams) in the construction of overlay trees. It is unlikely that a single Application Layer Multicast protocol will satisfy all classes of applications. Instead, we conjecture that different ALM protocols will coexist, each targeting a different set of applications. Just as our ESM protocol is designed for multisender teleconferencing applications involving high-bandwidth flows, another protocol may target applications with lowbandwidth flows but require reliability and/or quality of service adaptation. Providing quality of service and reliability for IP Multicast has been an interesting and difficult problem and this will probably be the case for ALM as well, though ALM has the promise of adapting wellknown unicast solutions to the overlay multicast case. 9 Conclusion Multicasting is a critical requirement for many DVE applications, but the deployment of IP Multicast network components has been slow, especially in reaching home Internet users. Recent efforts have investigated the merits of pushing multicasting functionality from the network layer to the application layer, with it thus being immediately deployable over the Internet. Such efforts have so far primarily targeted single-source applications with large receiver sets such as video-ondemand or live broadcasts. This paper introduces application layer multicasting as it relates to DVE and distributed CVE applications, highlights the shortcomings of current protocols in addressing such applications and presents our end system multicasting protocol designed for a particular class of CVE applications: multisender teleconferencing. It is our hope that this work will fuel other research endeavors in providing application layer multicasting for DVE and distributed CVE applications. References Aharoni, E., & Cohen, R. (1998). Restricted dynamic Steiner trees for scalable multicast in datagram networks. IEEE/ ACM Transactions on Networking 6(3), Baker, H. H., Bhatti, N., Tanguay, D., Sobel, I., Gelb, D., Goss, M. E., et al., (2003). Computation and performance issues in Coliseum, an immersive videoconferencing system. Proceedings of 2003 ACM Conference on Multimedia (MM 03),

14 276 PRESENCE: VOLUME 13, NUMBER 3 Balakrishnan, H., Rahul, H. S., & Seshan, S. (1999). An integrated congestion management architecture for internet hosts. Proceedings of 1999 ACM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM 99), Banerjee, S., Bhattacharjee, B., & Kommareddy, C. (2002). Scalable application layer multicast. Proceedings of 2002 ACM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM 02), Banerjee, S., Kommareddy, C., Kar, K., Bhattacharjee, B., & Khuller, S. (2003). Construction of an efficient overlay multicast infrastructure for real-time applications. Proceedings of 2003 IEEE Conference on Computer Communications (INFOCOM 03), Barrus, J. W., Waters, R. C., & Anderson, D. B. (1996). Locales: Supporting large multi-user virtual environments. IEEE Computer Graphics and Applications, 16(6), Bauer, F., & Varma, A. (1995). Degree-constrained multicasting in point-to-point networks. Proceedings of 1995 IEEE Conference on Computer Communications (INFOCOM 95), Chawathe, Y., McCanne, S., & Brewer, E. A. (2000). RMX: Reliable multicast for heterogeneous networks. Proceedings of 2000 IEEE Conference on Computer Communication (IN- FOCOM 00), Chen, S., Gunluk, O., & Yener, B. (2000). The multicast packing problem. IEEE/ACM Transactions on Networking, 8(3), Chu, Y., Rao, S. G., Seshan, S., & Zhang, H. (2002). A case for end system multicast. IEEE Journal on Selected Areas in Communication: Networking Support for Multicast [Special Issue] 20(8), CUWorld (2003). Videoconferencing Software. Retrieved April 30, 2003, from Das, T. K., Singh, G., Mitchell, A., Kumar, P. S., & McGee, K. (1997). NetEffect: A network architecture for large-scale multi-user virtual worlds. Proceeding of 1997 ACM Symposium on Virtual Reality Software and Technology (VRST 97), Deering, S., & Cheriton, D. (1990). Multicast routing in datagram Internetworks and extended LANS. ACM Transactions on Computer Systems, 8(2), Deering, S., Estrin, D., Farinacci, D., Jacobson, V., Liu, G. G., & Wei, L. (1994). An architecture for wide-area multicast routing. Proceedings of 1994 ACM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM 94), Diot, C., Levine, B. N., Lyles, B., Kassem, H., & Balensiefen, D. (2000). Deployment issues for the IP multicast service and architecture. IEEE Network Magazine, 14(1), Eriksson, H. (1994). MBONE: The multicast backbone. Communications of the ACM, 37(8), Francis, P. (1999). Yoid: Extending the multicast Internet architecture. htmlroot.html Frécon, E., Greenhalgh, C., & Stenius, M. (1999). The DIVEBone An application-level network architecture for Internet-based CVEs. Proceedings of 1999 ACM Symposium on Virtual Reality Software and Technology (VRST 99), Funkhouser, T. A. (1995). RING: A client-server system for multi-user virtual environments. Proceedings of 1995 SIGGRAPH Symposium on Interactive 3D Graphics, Greenhalgh, C., & Benford, S. (1995). MASSIVE: A collaborative virtual environment for teleconferencing. ACM Transactions on Computer Human Interactions 2(3), Greenhalgh, C., Benford, S., & Reynard, G. (1999). A QoS architecture for collaborative virtual environments. Proceedings of 1999 ACM Conference on Multimedia (MM 99), Greenhalgh, C., Purbrick, J., & Snowdon, D. (2000). Inside MASSIVE-3: Flexible support for data consistency and world structuring. Proceedings of 2000 ACM Conference on Collaborative Virtual Environments (CVE 00), Hosseini, M., & Georganas, N. D. (2003). Design of a multisender 3D videoconferencing application over an end system multicast protocol. Proceedings of 2003 ACM Conference on Multimedia (MM 03), Konemann, J., & Ravi, R. (2003). Primal-dual meets local search: Approximating MST s with non-uniform degree bounds. Proceedings of 2003 ACM Symposium on Theory of Computing (STOC 2003), Kortsarz, G., & Peleg, D. (1998). Generating low-degree 2-spanners. SIAM Journal on Computing 27(5), Kwon, M., & Fahmy, S. (2002). Topology-aware overlay networks for group communication. Proceedings of 2002 ACM International Workshop on Network and Operating System Support for Digital Audio and Video (NOSSDAV 02), Levine, B. N., Crowcroft, J., Diot, C., Garcia-Luna-Aceves,

15 Hosseini and Georganas 277 J. J., & Kurose, J. F. (2000). Considerations of receiver interest for IP Multicast delivery. Proceedings of 2000 IEEE Conference on Computer Communication (INFOCOM 00), Macedonia, M., Pratt, D., & Zyda, M. (1994). NPSNET: Network software architecture for large scale virtual environments. Presence: Teleoperators and Virtual Environments, 3(4), Macedonia, M., Zyda, M., Pratt, D., Brutzman, D., & Barham, P. (1995). Exploiting reality with multicast groups. IEEE Computer Graphics and Applications, 15(5), Malouch, N. M., Liu, Z., Rubenstein, D., & Sahu, S. (2002). A graph theoretic approach to bounding delay in proxyassisted, end-system multicast. Proceedings of the Tenth International Workshop on Quality of Service (IWQoS 02). Mathy, L., Canonico, R., & Hutchison, D. (2001). An overlay tree building control protocol. Proceedings of the Third International Workshop on Networked Group Communication (NGC 01), Padmanabhan, V. N., Wang, H. J., Chou, P. A., & Sripanidkulchai, K. (2002). Distributing streaming media content using cooperative networking. Proceedings of 2002 ACM International Workshop on Network and Operating System Support for Digital Audio and Video (NOSSDAV 02), Pendarakis, D., Shi, S., Verma, D., & Waldvogel, M. (2001). ALMI: An application level multicast infrastructure. Proceedings of the Third Usenix Symposium on Internet Technologies and Systems. Ratnasamy, S., Handley, M., Karp, R., & Shenker, S. (2001). Application-level multicast using content-addressable networks. Proceedings of 2001 International Workshop on Networked Group Communication, Ravi, R., Marathe, M. V., Ravi, S. S., Rosenkrantz, D. J., & Hunt, H. B. (2001). Approximation algorithms for degreeconstrained minimum-cost network-design problems. Algorithmica 31(1), Reynard, G., Benford, S., & Greenhalgh, C. (1998). Awareness driven video quality of service in Collaborative Virtual Environments. Proceedings of 1998 ACM Conference on Human Factors in Computer Systems (CHI 98), Shi, S. Y., & Turner, J. S. (2002). Routing in overlay multicast networks. Proceedings of 2002 IEEE Conference on Computer Communication (INFOCOM 02), Singhal, S., & Zyda, M. (1999). Networked Virtual Environments: Design and implementation. New York: Addison- Wesley. Taylor, M. J., & Rowe, S. M. (2000). Gaze communication using semantically consistent spaces. Proceedings of 2000 ACM Conference on Human Factors in Computer Systems (CHI 00), Tran, D. A., Hua, K. A., & Do, T. (2003). ZIGZAG: An efficient peer-to-peer scheme for media streaming. Proceedings of 2003 IEEE Conference on Computer Communication (INFOCOM 03), Vertegaal, R., Weevers, I., Sohn, C., Cheung, C. (2003). GAZE-2: Conveying eye contact in group video conferencing using eye-controlled camera direction. Proceedings of 2003 ACM Conference on Human Factors in Computer Systems (CHI 03), Zhang, B., Jamin, S., & Zhang, L. (2002). Host multicast: A framework for delivering multicast to end users. Proceedings of 2002 IEEE Conference on Computer Communication (INFOCOM 02), Zhang, R., & Hu, Y. C. (2003). Borg: A hybrid protocol for scalable application-level multicast in peer-to-peer networks. Proceedings of 2003 ACM International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 03), Zhuang, S. Q., Zhao, B. Y., Joseph, A. D., Katz, R. H., & Kubiatowicz, J. D. (2001). Bayeux: An architecture for scalable and fault-tolerant wide-area data dissemination. Proceedings of 2001 ACM International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 01),

16 278 PRESENCE: VOLUME 13, NUMBER 3 Appendix Global Inputs: A fully connected graph G (V,E) and a maximum out-degree outd_max(v) for all nodes v V. Rs: a target set for each source s Ts: tree rooted at a source s and spanning its targets Softjoin(s,t) if t Ts: if InsertNode(s,t) yields no solution: SoftjoinAll() SoftjoinAll() Define A {s V outd_max(s) Rs } Define B {s V s A} for all s A: for each v Rs: add edge {s,v} to Ts for all s B: outd(s) outd_max(s) for all s B: in descending order of (outd_max(s)- Rs ): outd(s) 0 avail_outd outd_max(s) (outd_max(i)-outd(i)), for all i Rs if avail_outd Rs AND Ri outd_max(i), for all i V: find r Rs, r s with (outd_max(r)- outd(r)) 1: Rs {r} while(outd(w) outd_max(w), for w Ts): add edge{p,u} to Ts, where: p Ts with max(outd_max(r)-outd(r)) u Rs with max(outd_max(r)-outd(r)) Rs- {u} if Rs Ts for any s V: InsertNode(s,t) for any t Rs, t Ts attach(s,t,c) if c Ts: add edge {c,t} to Ts else: p child of s add edge {p,c} to Ts replace edge {s,p} with {s,c} in Ts InsertNode(s,t) for each c V with outd_max(c) outd(c): if c Rs: attach(s,t,c) else (unless have attempted c,s pair before): relocate_spare_outdegree(c,s) Relocate_spare_degree(c,s) if for all d Rs, can_relocate(c,d) does not yield solution: for all a Rs, a c: if can_relocate(c,a) yields solution: (unless have attempted a,s pair before): relocate_spare_degree(a,s) can_relocate(c,d) for any e V: if c Te AND d Te AND Te 2: Transfer a load of d to c Softleave(s,t) if number of children of t in Ts 1: keep target t as reflector else: remove edge {parent_of_t,t} from Ts add edge{parent_of_t,child_of_t} to Ts if child_of_t exists