Integrating Commercial High-Capacity MPLS WAN Services in Critical Operational Environments

Transcription

1 SpaceOps 2010 Conference<br><b><i>Delivering on the Dream</b></i><br><i>Hosted by NASA Mars April 2010, Huntsville, Alabama AIAA Integrating Commercial High-Capacity MPLS WAN Services in Critical Operational Environments Ernesto Dölling, Manfred Bertelsmeier ESA-ESOC, OPS-ECT, Robert-Bosch-Str. 5, D Darmstadt Eden Warhurst Vega Deutschland GmBH & Co KG, Europaplatz 5, D Darmstadt Downloaded by on July 4, DOI: / The requirements on performance, availability and security imposed on the ground segment by new space missions have set a very high standard for the underlying ICT infrastructure services, especially in the area of Wide Area Networks (WAN). The classical WAN solutions used by ESA, for several years based on the rental of commercial clear channels, presented some compliance problems with requirements for increased capacity, especially in terms of flexibility, scalability and cost. This paper describes in detail the study made by ESA/ESOC on the applicability of MPLS services to the transport of highly critical operational data in a typical ground segment network, the issues and challenges found during this study, and how these were resolved, finally leading to the design of a network architecture with an enhanced convergence concept that integrates commercial MPLS services into the critical ground segment WAN. The paper also covers details regarding the implementation of this architecture inside the ESA ground segment network, its successful role in supporting the Herschel and Planck missions, and its scalability in view of upcoming missions with yet higher data rates, such as GAIA. I. Introduction ESA operates a global system of ground stations known as ESTRACK, the ESA Tracking Network. Communications services for all ESTRACK stations are provided by a network known as OPSNET. Dedicated to space mission operations, OPSNET has to fulfill the needs of the ESA mission model with a right-sized design at the most cost-effective procurement conditions 1. Two design parameters are of high leverage in this respect: network link topology and network link bandwidth. Prior to missions such as Herschel-Planck and GAIA, the highest mission-specific data rate requirements to be handled by OPSNET were below 300 kbps. Other services supported via OPSNET, such as voice teleconferencing or remote monitoring and control (M&C) of tracking stations, consume bandwidths of some tens of kbps. With data rates as moderate as that, it was for several years sufficient for OPSNET to deploy telecom products from the first level of the Synchronous Digital Hierarchy (SDH), which for the European variant of the SDH is referred to as E1. E1 links have clock rates of up to 2 Mbps. They were procured as clear channels, interfacing directly to IP routers managed by ESOC. This corresponded well to some essentials of the OPSNET service management philosophy, in this case the potentials for change flexibility and rapid diagnostics. Prices of full E1 links soon became so competitive that fractional E1s were no longer attractive, and in essence, E1 links became the norm in OPSNET as far as capacity was concerned. Further economies were achieved by optimization of the network topology. Advantage was taken from the close neighborhood of traditional and deep space tracking stations in Australia (Perth, New Norcia) and Spain (ESAC, Cebreros). Before the integration of deep space terminals in ESTRACK, OPSNET was a pure star network, with one pair of WAN links between ESOC and each of the ground stations. When the first deep space terminal was commissioned at New Norcia in Australia, the sites ESOC, Perth, and New Norcia were interconnected by a ring of E1 links. Instead of four transoceanic links as in a classical redundant star, only two were deployed, enabling drastic savings in international link rental. To achieve full redundancy, even the smallest potential common point of failure was avoided on the physical level. The availability of two diverse data paths per terminal was then achieved on logical level via suited IP routing configurations. - With the opening of the second ESA deep space terminal at Cebreros, Spain, the ring topology was repeated with Cebreros, ESAC, ESOC. The related topology of OPSNET is depicted in Fig.1. 1 Copyright 2010 by ESA. Published by the, Inc., with permission.

2 Downloaded by on July 4, DOI: / The deep space terminals in close vicinity to traditional ESTRACK sites made possible the network rings, but deep space missions also brought a new challenge to OPSNET, which exceeded single E1 bandwidth limits: Delta DOR. This ranging technique, with two antennae acting as a very large base interferometer, requires the transfer of very high data volumes from the terminals in a finite time. This need was found to be well in excess of what a single E1 link could support on top of routine TT&C support. Unfortunately, DeltaDOR measurements are required far too seldom to justify that the entire Figure 1: OPSNET Topology Using Clear Channels routine networking is sized for it. Nonetheless, a solution was found for which E1s were sufficient, based on the dual IP paths on the rings. These data paths, designed as hot redundancy pair for TT&C, of which at any time only one is used, were modified to act, for DeltaDOR - and only for DeltaDOR - in load share mode, thus yielding a combined capacity well in excess of 3 Mbps 2. - With this modification, however, the limits of bandwidth supportable in a network with single E1 trunks had finally been reached. II. Requirements for Ground Segment Communications and Possible Solutions A. Key Ground Segment Requirements The beginning of 2008 brought the decisive challenge to the OPSNET: space missions with telemetry data rates 3 beyond the 2Mbps limit, like GAIA and Herschel-Planck, and making extensive use of the Deep-Space antenna in New Norcia, Australia. It was soon evident that the classical OPSNET model, designed to make use of clear-channel leased lines, was going to be difficult to sustain. Scaling up would have meant to go from E1 (2 Mbps) to the next commercially available level of the Synchronous Digital Hierarchy (SDH/SoNET). This being 34 Mbps (E3), such high bandwidth was not going to be an affordable solution, the more so as the costs reach prohibitive levels if E3 were deployed on trans-continental level. Several alternatives appeared as possible solutions to this capacity-driven challenge, such as: Virtual Private Networks (VPN) over Internet, Multilink bundles, and WAN services based on Multi-Protocol Label Switching (MPLS). However, the OPSNET has to satisfy a number of other requirements than transferring volumes of data at a given bit rate. Without doubt, availability and quality of service are main requirements also to be taken into account. Availability of a network can be measured in a number of ways, however one indicator typically drives the designs of the ESA OPSNET: the time to restore connectivity after a failure, the TTR. The TTR has a very significant impact on certain applications using the OPSNET, like the transfer of real-time spacecraft telemetry, or the man-machine interface for remote monitoring and control (M&C) of all systems in a ground station. An interruption of service on a particular link during critical operations could cause irrecoverable loss of information, or trigger a tedious restart of application which introduces an unnecessary delay during critical operations. Both effects are undesired. The OPSNET design threshold therefore is a TTR of maximum 10 seconds. This is well achievable in pure data centre environments 4, but on a network spanning the globe it is a serious challenge. Quality of Service (QoS) in data networks refers to the ability of the network to ensure for a particular data-flow a stable set of network parameters (typically throughput or delay), regardless of the actual network load or congestion. In the case of the ESA OPSNET this requirement is absolutely mandatory. Without QoS, low priority flows like file transfers (routine small or high volume bulk transfers) might easily cause interruptions to real time critical flows, which are typically of low volume, but with high operations impact. B. Potential Solutions Internet VPNs For many scenarios requiring long-distance IP based multi-point networks, a virtual private network (VPN) on top of the Internet, commercial or academic, is an attractive solution. Based on secured encrypted channels between end-points, a VPN is a relative inexpensive network. However, space missions have stringent service requirements 2

3 Downloaded by on July 4, DOI: / that are in conflict with this technology as main underlying infrastructure: the lack of an SLA with the required availability and performance guarantees (QoS). Therefore the Internet VPN option as backbone baseline was discarded. Nonetheless, some OPSNET peering sites with less critical use are connected using a commercial product with an SLA as prime, and an Internet VPN link as back-up. Multilink Bundles This solution is a capacity increase based on clear channels, and was considered fairly simple to deploy. The principle is to bundle as many physical links as required in order to obtain a single logical link with the desired speed. The protocol to implement such solution is called Multilink PPP (M-PPP), an expansion of the standard for serial lines, PPP (Point-to-Point Protocol). This technique is well known in the world of ISDN, were it is common practice to bundle a number of 64 kbps B-channels to obtain aggregate links with higher bandwidths. The technique is however not free of challenges, especially when trying to bundle links over long distances. Different delays presented by different physical links introduce severe jitter (variable delay) to network packets, or even cause packet reordering, affecting in particular IP streams not using the transmission control protocol TCP. For voice over IP (VoIP), e.g., the reception of disordered packets represents a completely unacceptable loss of quality. M-PPP also has an economic drawback: poor scaling. The cost of the solution increases linearly with the number of links, yielding in essence no economy of scale. In addition, the segmentation / assembling issues described above grow with the number of links, turning virtually impossible in practice to bundle more than eight links. MPLS VPN Services The Multi-Protocol Label Switching (MPLS) technology was born as an attempt to bring IP networks into the world of the telecom service providers, or telecom carriers. At the beginning of the 90's, the effort of the telecom carriers for data networks was focused on making the ATM technology as fast and reliable as possible. The core of the ATM technology was its ability to quickly process data cells in ATM switches. Information was fragmented into small cells (up to just 56 bytes) and then transmitted over logical paths to the end destination. However in the late 90's, with the explosion of technological advances like ASIC, TCAM and CAM-based switching, the need for splitting data into small cells for fast switching became obsolete, and the logical step was made: instead of switching ATM cells, why not to directly switch IP packets? The result was MPLS. MPLS has evolved in the last 10 years into a reliable technology which allows telecom carriers to transport traffic over a common data network which can be securely shared among several customers. Several flavors of MPLS services exists in today s market, from the classical IP transport over L3-VPNs (RFC 4364), up to the pseudo wire solution, or L2VPN, which allows to transport any protocol over MPLS (RFC 4447). Though being IP based like its cousin Internet, MPLS inherits from ATM the service oriented architecture and the ability to guarantee, among other characteristics, service levels (availability and performance). A typical MPLS network is shown in Figure 2. Its main part is a core or backbone network. This backbone is an IP network managed by the Service Provider on behalf of several customers. The traffic of an individual customer is encapsulated into MPLS packets. These carry an additional MPLS label which uniquely identifies the path that the traffic needs to follow inside the network. Individual customers connect to this network via dedicated access links to the closest Point of Presence (PoP) of the provider. An end-to-end path between two customer sites then consists of three segments: access link between source and core, MPLS path inside the core, and access link between core and destination. On physical level, the access links are usually clear channels out of the SDH, e.g. E1, fractional or bundled E1, E3. Important is the fact that the access links are short distance links. Only on this level does an individual customer have to bear the cost of potential excess capacity on clear channels (e.g. for a traffic of 5 Mbps on a 34 Mbps E3 link), but not on the long haul segment in the core. On this part, the share of many MPLS paths between many customers yields an attractive economy of scale. Figure 2: OPSNET based on an MPLS network. 3

4 III. Integrating MPLS Services in the Network Design Based on the previous analysis ESOC decided to implement the new OPSNET architecture based on MPLS technology. The next task was to ensure that the overall design was compliant with both operational and mission requirements. Some of the more important requirements that needed to be adhered to, and which also become major factors in determining the final network design included the following: Interoperability with current network topology. Network convergence less than 10 sec in the instance of primary path failure. Independent primary and secondary paths at both physical and logical levels. Ability to have site-to-site connectivity (not just hub-spoke topology) within the MPLS cloud. To fulfill the above requirements, in-depth technical discussions, both internally and with the provider, were held. The provider committed to deliver both independent primary and secondary paths at both physical and logical levels along with the ability to have site-to-site connectivity between the remote sites, thus ensuring that two of the main requirements were fulfilled. This left ESA with the decision on how routing information and IP datagrams would be sent across the MPLS/VPN backbone. Certain options were presented by the provider. Downloaded by on July 4, DOI: / A. Physical Topology and Definition of Demarcation Line One of the first topics to be defined for integrating MPLS based services into a customer network is the physical topology. For the case of OPSNET, where critical mission operations need to be supported, a fully redundant topology is a must. It involves links and routers, namely CE routers - customer edge routers, the MPLS routers located in customer premises, and PE routers - provider edge routers, those located at the interface between access link and core. The MPLS core network thus needs to be accessed via two distinct PE devices, typically located in different PoPs. This is mirrored by different CE devices, which make use of a suited underlying infrastructure (e.g. links using different local carriers, or dual protected SDH rings as used for the telecom provider PoP at ESOC 4 ). Once the physical layout has been delineated, the network management demarcation line between MPLS provider and customer needs to be defined. Two options exist for this: - Customer furnished and managed CE devices. - Service Provider furnished and managed CE devices. Both alternatives have pro's and con's, however the latter brings the decisive advantage: a LAN interface constitutes a cleaner demarcation of responsibilities, is easier to troubleshoot, and gives the service provider full control over the end-to-end WAN service. Consequently ESA decided for a provider managed CE router solution. The setup used for OPSNET is shown in Fig. 3. ESA managed routers are used for interfacing to the provider managed CE device. This topology could be seen as overly hardware intense, as it is also possible to allow the provider to connect directly to the LAN (thus avoiding the usage of an extra device in the chain). However the justification is clear when one considers the need of performing customer controlled change management at the border of the network, like QoS (see Chapter IV C) and security (access control, encryption), features that are rarely available from providers with guaranteed turn-around times for changes. Figure 3: MPLS interface physical topology B. Quality of Service Potential of MPLS Inside an MPLS/VPN network, the service provider typically offers various classes of services (CoS), The CoS support prioritization of data packets within the cloud, of interest in particular for delay sensitive traffic flows such as VoIP, telemetry and telecommand. The CoS typically offered by any commercial MPLS provider include the following: PRIORITY: reserved for traffic vulnerable to delay and jitter constraints such as real-time voice, video, etc. Delay and jitter are minimized in this class by strict bandwidth reservation and prioritization. GOLD: reserved for business critical traffic. This class offers a guaranteed throughput with zero packet drop probability in case of congestion, and therefore has a guaranteed bandwidth not lower than a set minimum. 4

5 Downloaded by on July 4, DOI: / SILVER: used for traffic from interactive applications. This class offers a bandwidth guaranteed service, but with higher packet drop probabilities in case of congestion compared to the GOLD service. BRONZE/BEST-EFFFORT: used for non referenced traffic or traffic insensitive to delays or high packet drops (e.g. mail, file transfers). This class uses the remains of bandwidth left by other classes, and therefore has no bandwidth guaranteed. The implementation of different CoS is based on the Differentiated Services Code Point (DSCP, DiffServ) feature (RFC 2475), used to categorize and allocate network resources. The DiffServ architecture defines a field (called DS field) within the header of an IP packet to carry information how per-hop behaviour (PHB) decisions are to be handled within the MPLS network. Packets need to be marked prior to entering the MPLS VPN network on the edge routers by setting the DS field to a pre-defined value. The DS value of Figure 4: DSCP used for CoS in MPLS networks. the IP packet is then read by the CE/PE router and copied into the Type of Service (ToS) field of the MPLS label and sent through the cloud. Depending on the marking, the packets will be queued differently as they traverse the MPLS network. Delay sensitive packets such as VoIP will be given priority over less delay sensitive packets. C. Issues with Routing Once the interconnection to and the use of the MPLS service has been defined, a selection of the method to exchange routing information needs to be done. The MPLS VPN service needs to know which network prefixes are reachable at each customer location. The exchange of network prefix information can be done via static routes or via dynamic routing protocols, like Open Shortest Path First (OSPF) or Border Gateway Protocol (BGP). The dynamic option has the obvious advantage to allow the customer to add/remove prefixes at will, while static route changes require agreement and manual configuration from the provider side. OPSNET uses OSPF as Interior Gateway Protocol (IGP), therefore it appeared reasonable to use it also for exchanging routing information with the service provider. This however did not result in the simplest option. Lab analysis showed the solution used in OPSNET was not compatible with the one implemented by the provider. Initially the interaction with MPLS impacted the OSPF backbone configuration and the assurance of loop-free topologies by OSPF. This was overcome by a minor change to the OSPF topology. The last point left to optimize was the speed of convergence. By using circuit protection methods, the lab results were very promising (see Chapter V), however the provider implementation in the field fell severely short of the needs. A detailed analysis showed the reasons behind the poor convergence time. When a topology change occurs in the network (e.g. a link is going down), all routers in the network need to re- protocols. In learn the path to avoid the failure. This learning process is dynamically controlled by a set of routing the case of MPLS networks, the routing process involved is called Multi-Protocol Border Gateway Protocol (MP- BGP). MP-BGP is run by the service provider between the PE routers and a central router, called Route Reflector. Each PE holds a dedicated routing table for each CE that connects to it, called Virtual Routing Forwarding (VRF). This VRF feature allows Service Providers to transport the traffic and routing information of multiple customers over the MPLS backbone in a complete separate and secure way. The protocol used between the PE and the CE depends on the implementation, and is typically either BGP or OSPF. Figure 5 illustrates principle processes of an MPLS network which are involved as import/export/exchange processes in the overall convergence mechanism. - P1, P7: Information exchange between the PE and CE after a change in the network has occurred (timers T1, T7). - P2: Import the local routing information into the corresponding VRF routing table (timer T2). - P3: Reception of local routes into the MP-BGP table of the PE Router (timer T3). - P4: Propagation of changes in the MP-BGP routing table to MP-BGP peers (timer T4). - P5: Reception of advertised routes into MP-BGP Table on the PE Router (timer T5). - P6: Import of Newly Received Routes into Local VRF s Routing Table (timer T6). - P8: Processing of incoming updates by the CE Route (timer T8). These processes are fully in control of the Service Provider. Their timers are typically kept at conservative values to ensure maximum network stability. T1, T4, T6 and T7 typically make up 90% of the total delay. 5

6 Downloaded by on July 4, DOI: / The maximum theoretical time for a change in the network then to be propagated from a Local CE to a remote CE is given by adding up the times required for each of the processes to be carried out. Max time= T1 + 2.T4 + T6 + T7 + x, where x = T2 + T3 + T5 + T8, Max time= x = 85 sec + x sec. The maximum theoretical TTR (or convergence) of at least 85 seconds was confirmed by the service providers, which stated that typical times involving an MPLS core expanding from Europe to Australia could be up to several minutes. These preliminary results were clearly in opposition of the main requirement for the OPSNET to be able to automatically restore service after an incident within 10 seconds. A possible way to overcome this limitation is described in the following section. Figure 5: Processes involved in the convergence of MPLS networks IV. Optimizing the Time to Restore (TTR) The challenge was to find a solution that would permit ESA to use a commercial of-the-shelf solution, at a very competitive pricing, but in an adaptation compatible with all requirements of mission operations. A. A Solution Using GRE Tunnels For overcoming the slow convergence issue, consideration was first given to non-standard vendor solutions. A path can be protected against failures by setting up in advance an alternate diverse path. The technique can be used inside the MPLS core to restore services in less than 50ms, similar to protected paths in SDH/SoNET. Relatively new and vendor specific, this option was not available from all service providers, not in all countries, and only applicable to failures inside the backbone. A different technique was therefore studied: a logical overlay over the MPLS network, which would allow to abstract the complexities of the MPLS cloud, and to run a separate routing protocol on top of it, fully in control of the customer. This logical overlay would be achieved via the Generic Encapsulation Protocol (GRE). GRE (defined in RFC 1702) is a tunneling protocol which encapsulates network layer packet types over an IP tunnel. By using GRE we were able to establish virtual point-to-point links, over the carrier s MPLS VPN network between the ESOC managed edge routers. GRE would encapsulate the customer traffic on the transmitter side, and decapsulate it on the receiver side. Any customer traffic would be transparently transported inside the GRE encapsulation (also called GRE tunnel), inclusive routing protocols if required. The solution concept is shown in Figure 6, already depicted for a redundant set of tunnels. The solid resp. dotted lines represent encapsulated traffic of customer routers. All the provider network sees is the transport of IP-GRE packets between pairs of end devices. The setup allows for the customer to put any desired traffic inside the GRE encapsulation in a transparent way. One of the main advantages to this approach is that, apart from some management IP addresses, the only IP prefixes that the carrier needs to advertise into the MPLS/VPN cloud are the endpoint IP addresses of the ESOC managed Figure 6: GRE Tunnels as Prime / Secondary Paths edge routers. These endpoint addresses are statically routed into the MPLS/VPN cloud. 6

7 Downloaded by on July 4, DOI: / How can this GRE tunneling help on convergence? The tunnels are used not only to transport customer traffic, but also to run an interior routing protocol (OSPF in the case of OPSNET). The setup is shown in Fig 6. Two logical GRE tunnels are run over the MPLS network, each initiated and terminated in a different router to ensure physical redundancy. From the IGP point of view, the local and remote router pairs form a direct adjacency, pretty much as they were directly connected by a physical interface. The IGP adjacencies and the timers involved in it are fully in control of the customer. It is perfectly possible to fine tune the IGP to achieve fast restoration between prime and backup GRE tunnels. The independence of the two tunnels can be very easily achieved on the access network, where the tunnels are directly mapped onto different physical paths. However inside the MPLS core, the service provider has to support some kind of traffic engineering in order to ensure that mutual-redundant GRE tunnels will use actually different active paths inside the MPLS backbone cloud. Once the GRE tunnels were set up, the OSPF routing protocol was enabled as IGP across the tunnel interfaces and OSPF peering session were established directly between ESOC and the remote sites. This configuration allowed the MPLS VPN cloud to be transparent to the OPSNET and provided a clear routing domain demarcation line between ESOC and the carrier. It also met the requirement for interoperability with the current network topology and allowed for a seamless migration between the legacy E1 OPSNET trunks and the MPLS VPN cloud. To ensure that the network convergence was compliant with the given mission requirements some additional fine-tuning was required on the GRE tunnels. Firstly, GRE keepalives were enabled on the tunnel interfaces. This was required because by default, GRE does not make use of a keepalive mechanism and the line protocol status on a GRE tunnel will remain up even if the remote end of the tunnel is unreachable. By enabling keepalives, each tunnel endpoint constantly sends and acknowledges keepalive packets with the remote endpoint. If multiple packets are not acknowledged, the line protocol status on the tunnel will change to down. Once the tunnel is down, the OSPF routing protocol will instantaneously remove the routing information received over that tunnel from its routing table and re-route via an alternate path (in our case the secondary tunnel). The OSPF hello interval timers on the tunnel interfaces were also tuned to expedite path convergence. B. Issues and Benefits of Using GRE Tunnels 1. Issues The solution provided by GRE tunnels comes however not for free. The GRE encapsulation needs a GRE header, 24 Bytes long. In practical terms, this header reduces the maximum transfer capacity of an IP packet by 24 Bytes. This may sound not a big issue when one compares the total 1500 Bytes default Maximum Transfer Unit (MTU) supported by MPLS networks, however it creates an impact to IP flows attempting to use the maximum capacity of the network. When a router receives an IP packet of greater size than the supported MTU, the protocol will fragment the packet, and subsequently reassemble it at the receiver end. This is fine until one considers that the host at the far end will need to store the fragments of the packet until all of them are received and can be reassembled to the full packet. This causes extra load on the hosts CPU and memory, and adds extra latency. Additionally, fragmentation theory is not always followed very well by practice, showing more than a challenge to get it working properly between IP stacks of different manufacturers. To avoid unnecessary fragmentations, a technique is available called Path MTU discovery. Fully described in RFC 1191, the mechanism works as follows: before establishing a connection to a particular destination, the sending host will probe the network by sending a packet with size set up to the MTU of its local interface, but with the don't fragment (DF) bit set on. By doing so, if any router in the path from source to destination finds that the pack et is too big to be transmitted with the available MTU, it will not attempt to fragment, in obeying to the DF indication, and therefore will simply drop the packet. While doing so, it informs the sender with an ICMP packet containing the information type 3 (unreachable), subtype 4 (needs fragmentation), indicating also the maximum supported MTU. When the sender receives the ICMP message it will know that the size attempted with that probe is not supported without fragmentation. After the Path MTU discovery process is finished, the sending host knows exactly the MTU supported by the network path between sender and receiver, and therefore can simply start transmission by using a packet size that does not require fragmentation. 2. Benefits These drawbacks of GRE tunnels are however outweighed not only by the fast convergence, but also by additional benefits that would otherwise not have been achievable. 7

8 Downloaded by on July 4, DOI: / * GRE overlay tunnels have the ability to conceal the routing information of the customer from the provider. This situation, also IP Header Data called ships in the night routing, is made possible by the customer IGP (in our case OSPF) established across the GRE tunnel and not directly with the provider. The routing information is encapsulated GRE Encapsulation with IPSec inside the GRE tunneling overlay and remains hidden from the provider. This simplifies greatly the interface to the service provider, thus defining a clear demarcation of responsibilities and failure Encrypted IP Header domains; should the GRE tunnels remain up, any issue in the GRE Header GRE Payload network is the responsibility of the customer. On the contrary, if a GRE tunnel is down, there is no doubt that the problem is inside the MPLS Encapsulation MPLS could, and therefore responsibility of the service provider. * GRE tunnels provide a simple basis for the encryption of data. This can be achieved by enabling the Internet Protocol Security Encrypted MPLS Label (IPSec) protocol suite and combining it with standards like MPLS Payload Advanced Encryption Standard (AES) over the existing tunnel interfaces. The encapsulation process is shown in Figure 7. The transparent tunneling provided by GRE is converted into a secure encrypted channel. The encryption/decryption process is performed Figure 7: Encryption of GRE tunnels on the tunnel endpoints and remains fully in control of the customer. Although IPSec has not yet been implemented on the OPSNET network, this feature can be enabled with minor effort. * When it comes to the validation and integration of the solution in the operational network, the GRE tunnels also provided a benefit: by leaving the logical tunnel overlay deactivated, it was possible to leave the MPLS network solution always-on and connected to the live OPSNET from day zero, monitored 7 days x 24 hours, before the final validation and transfer to operations was done. A simple procedure allowed the engineering team to activate or deactivate the GRE overlay within seconds, making the MPLS solution live and ready for validation, or sending it again dormant after validation. This situation permitted the usage of small maintenance windows during the validation phase, saving on scheduling time and team effort. * From the maintenance and operations point of view, the GRE tunnels also provided an advantage. Monitoring tunnels was like monitoring any other physical WAN interface. With this, all the complexity of operating an MPLS service was reduced to the same procedures used so far with leased lines. Even the topology maps from the monitoring tools were configured to show the physical and logical topologies of the MPLS in different maps, allowing the teams to detect and isolate physical failures in the access network independently from failures caused by the MPLS backbone. C. Implementation of Quality of Service One of the major concerns in supporting critical operation over a MPLS WAN service is ensuring that delay sensitive traffic flows such as TT&C and Voice are given the appropriate network resources within the MPLS network. The allocation of resources over IP networks is often referred to as Quality-of-Service (QoS). When using communication links such as clear channels, QoS resource management is simplified as the bandwidth of the clear channel is statically defined by the service provider (clock rate) and wholly dedicated to the customer. The customer can configure the appropriate QoS policy across the clear channel link and be confident that the policy is fully under his control. This is different when connecting to MPLS networks. Here the provider does not deliver a defined bandwidth, but commits to a particular Service Level (SLA) which defines a set of Classes of Service (CoS) contracted by the customer. In order to achieve the desired QoS for particular data flows of the customer, they must be mapped to the CoS menue of the provider. To this end, the ESA managed edge routers must perform three additional traffic conditioning tasks, shown in Fig. 8: Flow identification and marking with the proper DSCP value to ensure the data flows will be mapped and handled by the service provider within the correct CoS. Internal queueing, to compensate the lack of granularity offered by CoS and to ensure bandwidth reservations among different traffic flows that are served by the same CoS. A technology called class based weighted fail queueing is used for this purpose. Traffic shaping, to ensure that the traffic levels subscribed per each CoS are not exceeded, which would cause oversubscription of the MPLS VPN and subsequent packet dropping inside the MPLS network. 8

9 Figure 8: QoS Traffic Conditioning Downloaded by on July 4, DOI: / V. From Design and Optimization to Routine Use A. Testing and Validation in a Lab Environment To be able to study all tuning options as discussed in previous chapters, and in the end ensure that an MPLS VPN cloud could achieve full compliance with the requirements of the OPSNET, a validation network was set up in a laboratory environment, representative of the target topology. The validation network is shown in Fig. 9. It consisted of the following elements: An MPLS network representing the provider s network A WAN / LAN network representing the ESOC network A WAN / LAN network representing a tracking station network A network impairment device A Telemetry (TM) and Telecommand (TC) Client and Server Figure 9: Test and Validation Setup The MPLS network included a single MPLS core router, connected to Provider Edge (PE) routers which were in turn connected to redundant Customer Edge (CE) routers. Both the ESOC and tracking station networks included redundant LAN edge routers and LAN switches. A network impairment device was also connected in-line with the MPLS network to emulate network delays and packet loss, representative of the actual target topology. A TM/TC Server was connected to the LAN segment of the tracking station network and a TM/TC client was connected to the ESOC LAN segment of the of ESOC network. The validation network proved to be an invaluable tool as it allowed for various integration alternatives to be implemented and evaluated in an off-line environment without having any impact on operations. It was also possible to carry out end-to-end application testing and tuning across a representative target topology prior to migration. This gave both the mission user base of OPSNET and the ESOC network implementation teams confidence that the target topology and its tunable configuration parameters would be compliant with the needs of both users and maintainers of OPSNET. 9

10 B. Aligning Monitoring and Control Systems The OPSNET is monitored by the constantly staffed Computer Control Centre (CCC). The CCC provides roundand monitoring the-clock monitoring of computer and communications services and uses networks management tools such as HP OpenView, CiscoWorks and Netflow, to quickly identify and follow up incidents within the OPSNET MPLS network. The network management servers make use of the SNMP protocol to constantly poll the MPLS provider s CE and PE routers to detect possible issues. The decision to run overlay GRE tunnels has allowed our network management tools such as HP OpenView to map both, the logical point-to-point tunnels on the one hand and the physical CE and PE routers on the other. This allows for a simplified view of the MPLS network to be displayed on the network management master map, whilst at the same time providing the ability to go to higher resolution of the physical topology if needed. A Web based portal containing various management tools has also been made available by the MPLS provider. These tools allow for information such as bandwidth utilization, data drops and network outages to be viewed and exported in real-time. The portal also provides the ability to log and track tickets with the service provider. Additional tools to monitor the compliance of the SLA have also been configured on the OPSNET to constantly monitor the network and provide detailed reporting which are used to measure SLA parameters. Downloaded by on July 4, DOI: / C. Testing and Validation on Communications Infrastructure Level Prior to running application validation and testing across the provider tunnels, network level acceptance testing was carried out to ensure that the tunnels were in adherence to specifications described in the RFQ. The validation tests were performed during scheduled maintenance windows. During these windows the GRE tunnels were activated and in turn, all traffic was routed via the MPLS VPN. The results were very promising with the real MPLS network behaving as predicted in the laboratory tests and passing all the defined test criteria. The communications infrastructure validation tests were focused on the following areas: Link stability Tunnels were left activated with routing disabled across them and monitored for a period of 96 hours. Routing OSPF adjacency stability / Asymmetric routing / OSPF routing table stability was monitored for a period of 48 hours. Redundancy Simulation of primary network outage. Failover from primary tunnel to backup and recording of end-to-end time required to restore service. Network convergence was achieved as expected, within a limit of 10 seconds. Maximum Transmission Unit (MTU) MTU path discovery was tested through the GRE tunnel, working as expected with only an end-to-end MTU of 1460 Bytes caused by the GRE encapsulation. Round trip time (RTT) tested by sending thousands of ICMP packets and measuring the average round trip delay, verifying the specifications of the RFQ for each location and each redundant pair. Class of Service (CoS) FTP traffic was sent across the cloud and marked with a specific DSCP value to test the behaviour of various classes of service within the clouds, against the SLA. Throughput The MPLS links were loaded up to 99% capacity and the packet loss monitored over a 24hs period. Voice over IP (VoIP) Voice checks were carried out on local and remote voice loops. D. Validation on Mission Operations Level Once the communication infrastructure validation was successfully completed, a final validation of the end-to- validation end network was performed, transporting mission operations user traffic. As with the communications tests, these tests were carried out within a maintenance windows during times of non-critical activity at the given tracking station, having all traffic data routed via the new MPLS VPN service. The Mission Operations Level testing was focused on the following areas: SLE application testing - End-to-end Space Link Extension (SLE) protocol testing performed with simulation of the spacecraft in various data transfer modes (Online Timely, Online Complete, Offline, etc). M&C application testing - End-to-end Monitor and Control (M&C) application testing between the ESTRACK Control Centre (ECC) and the given tracking station. Applications tested included connectivity to the Station Computer (STC), etc. Redundancy testing - Simulation of a network outage in the MPLS backbone and measurement of the time to restore service for the above operational applications. 10

11 E. Operational Experience Some time after the validation on mission operations level was completed and the MPLS service was declared officially in operations, cases were detected where both paths in a redundant tunnel pair were failing simultaneously. The issue was immediately escalated to the service provider since it pointed to a non-compliance with the parameters established in the SLA, and upon investigation the root cause was found. Tunnels from ESOC to a particular destination, though diversely routed at the access level of the MPLS cloud, were following the same path internal to the MPLS backbone. This meant that any failure inside the backbone was affecting both prime and backup tunnels at the same time. In order to solve this issue, the service provider implemented alternate routing for the different tunnel circuits inside the MPLS cloud, thereby providing full end-to-end diversity, as requested by ESA. Since this difficulty was overcome, the service has run smoothly, supporting thousands of tracking hours for Herschel-Planck and all other ESA missions requiring services from the ESA Deep Space Stations and the ESTRACK terminals at Perth and Villafranca which use the same MPLS network. Downloaded by on July 4, DOI: / VI. Conclusion For the increase of OPSNET capacity for Australian and Spanish ESTRACK terminals, commercial high speed MPLS services with a rigorous path diversity topology have been deployed. In combining them with overlay GRE tunnels, the OPSNET communications service to its users has achieved an excellent performance in terms of near non-stop availability and throughput, with fail-over times between prime and alternate path in the order of seconds, and with optimal mappings of network transport capabilities to the types of traffic which rely on OPSNET. The demarcation between the network management domains of ESA and the MPLS service provider has been positioned between a so-called customer edge router under full responsibility of the MPLS provider and a LAN frontal router fully managed by ESA. In conjunction with the GRE tunnels, this layout leads to a very clear delineation of responsibility for management of configuration, incidents, and changes. The provider is in full control of all MPLS underlying infrastructure, ESA is in full control of how this infrastructure is used by OPSNET traffic. Common involvement of both parties is only needed when a major change has to be tackled, e.g. when higher capacity access links would need to be deployed, or essential parameters in the SLA between ESA and provider would need to change. As however the ESA mission model over a few years has been taken into account in designing these infrastructure elements, relative stability in these areas is assumed. The service management by the OPSNET maintenance and operations service contract has remained rather simple. The GRE tunnels are very similar to the previous traditional clear channels, with all associated ease of monitoring of e.g. routing or availability. The ability of the M&O service for on-line read visibility of provider equipment is better than in the past. The new setup is a good basis for deploying encryption of OPSNET transactions. Although not a mandatory requirement today, it may materialize in the future, as and when new projects need to be accommodated by OPSNET for which the ESA Security Directives would call for elevated data security. For changes to the way OPSNET users need to exploit the new network, no interaction with the provider is needed. The chosen service management demarcation thus also preserves all the agility expected from OPSNET by its user missions. This has also proven very beneficial when running the old and the new network in parallel, avoiding any mutual interference. Off-line verification and validation with real mission operations traffic had full independence and could be planned flexibly. The experiences with the overall concept of GRE on top of standard MPLS were positive enough to adopt this as the baseline for a few MPLS links deployed later in OPSNET, and for those still planned in the near future. Acknowledgments The handover of the new MPLS infrastructure to routine operations marked the completion of a collaborative effort of many parties. The authors gratefully acknowledge the excellent contributions of engineers in the contract for Operational ICT Infrastructure Evolution with Vega Deutschland, and of the account manager of the provider Stratos Global. Thanks are also due to colleagues of the Ground Facilities Operations Division and to the ESOC Reference Station engineers who supported or ran the intense infrastructure and mission validations, and to the OPSNET M&O team of Serco GmbH at ESOC and the M&O teams in the ground stations involved. 11

12 References 1 Bertelsmeier, M., Buscemi, G., New Communications Solutions for ESA Ground Stations, ESA Bulletin, No. 125, February 2006, pp M. Bertelsmeier, G. Buscemi et al, Information Technology Solutions for Delta-DOR Large Volume Data Transfers, SpaceOps 2006, June 19-23, 2006, Rome, Italy. 3 D. Heinzer, E. Doelling, Novel ICT Services and Systems Architecture for a Successful Herschel-Planck Mission, SpaceOps 2010, April 25-30, Huntsville, Ala, USA. 4 R. Hussock, M.Dadomo, High Availability Data Centre Design for Space Operations, SpaceOps 2010, April 25-30, Huntsville, Ala, USA. Appendix A Acron ym List Downloaded by on July 4, DOI: / AES Advanced Encryption Standard MPLS Multi-Protocol Label Switching ASIC Application-specific integrated circuit MTU Maximum transmission unit ATM Asynchronous Transfer Mode M&C Monitoring and Control BGP Border Gateway Protocol M&O Maintenance and Operations CAM Content Addressable Memory NNO New Norcia CBWFQ Class Based Weighted Fair Queueing NTU Network Terminating Unit CBQ Class Based Queueing OPSNET ESA s Operations Support Communications Network CE Customer Edge OSPF Open Shortest Path First CoS Class of Service PE Provider Edge COTS Commercial off the shelf PHB Per Hop Behaviour Delta DOR Delta Differential One-way Ranging PoP Point of Presence DF Don t Fragment PPP Point-to-point Protocol DS Differential Service P-MTU Path Maximum transmission unit ESOC European Space Operations Centre QoS Quality of Service ESAC European Space Astronomy Centre RFQ Request for Quotation ESTRACK ESA s System of Tracking Ground Stations RFC Request for Comment E1 1 st / lowest level of SDH as European Standard RTT Round Trip Time E3 3 rd level of SDH as European Standard SDH Synchronous Digital Hierarchy FTP File Transfer Protocol SLA Service Level Agreement GRE Generic Routing Encapsulation SLE Space Link Extension ICMP Internet Control Message Protocol SNMP Simple Network Management Protocol ICT Information and Communications Technology SONET Synchronous Optical Network IETF Internet Engineering Task Force SP Service Provider IGP Interior Gateway Protocol TCAM Ternary Content Addressable Memory IOS Internetwork Operating System TT&C Telemetry, Tracking and Commanding IP Internet Protocol TTR Time to Restore IPSec Internet Protocol Security ToS Type of Service ISDN Integrated Services Digital Network VoIP Voice over IP Kbps kilobit per second VPN Virtual Private Network LAN Local Area Network VRF Virtual Routing and Forwarding Mbps Megabit per second WAN Wide Area Network MLPPP Multilink PPP WFQ Weighted Fair Queueing 12