Evolution of QoS routing in the Internet

Evolution of QoS routing in the Internet Olivier Bonaventure Dept. Computing Science and Engineering Université catholique de Louvain http://www.info.ucl.ac.be/people/obo June 4th, 2004 Page 1 Agenda Routing basics The ToS approach The Integrated Services approach The Traffic Engineering approach Issues with QoS routing in the Internet Page 2

Architecture of a normal IP router Routing protocol Routing table The "best" paths selected from the routing table built by the routing protocols are installed in the forwarding table Control IP packets Forwarding Table Class. Shap. Pol IP packets Forwarding Class. Shap. Pol IP packets Page 3 Forwarding decision based on longest match Update of TTL and checksum fields in IP packets Interior Gateway Protocol (IGP) Internet routing Routing of IP packets inside each domain Only knows topology of its domain Domain4 Domain2 Domain1 Domain3 Page 4 Exterior Gateway Protocol (EGP) Routing of IP packets between domains Each domain is considered as a blackbox

Goal Intradomain routing Allow routers to transmit IP packets along the best path towards their destination best usually means the shortest path Shortest measured in seconds or as number of hops sometimes best means the less loaded path Allow to find alternate routes in case of failures Behavior All routers exchange routing information Each domain router can obtain routing information for the whole domain The network operator or the routing protocol selects the cost of each link Page 5 Static routing Three types of Interior Gateway Protocols Only useful in very small domains Distance vector routing Routing Information Protocol (RIP) Still widely used in small domains despite its limitations Link-state routing Open Shortest Path First (OSPF) Widely used in enterprise networks Intermediate System- Intermediate-System (IS-IS) Widely used by ISPs Page 6

Distance vector routing Page 7 R1 Principle C=3 C=5 R2 C=3 R3 Router configuration Cost associated with each link Each router sends periodically a distance vector containing, for each known prefix, : C=1 C=1 1. The IP prefix 2. The distance between itself and the destination The distance vector is a summary of the router's routing table Each router receives its neighbor's distance vectors and builds its routing table based on those vectors R5 R4 C=3 C=10 C=6 R6 Issues with distance vector routing How to deal with link failures? Routers should send their distance vector when they detect the failure of one of their links How to avoid the count-to-infinity problem? Utilize a non-redundant star shaped network Limit the maximum distance between routers For RIP, Split horizon 16! Router A does not advertise to router B the routes for which it sends packets via router B Split horizon with Poison reverse Page 8

Link state routing Page 9 C=3 R2 C=3 R1 Principle C=5 R3 C=1 R5 C=10 C=3 R6 C=1 C=6 R4 Each router builds link state packet containing its local topology Link state packets are created at regular intervals and when the local topology changes Link state packet is reliably flooded to all routers inside the domain Each router knows the complete domain topology Computes routing tables by using Dijkstra The best path is the path with the smallest cost Agenda Routing basics The ToS approach The Integrated Services approach The Traffic Engineering approach Issues with QoS routing in the Internet Page 10

The ToS byte ToS : Type of Service Octet defines the relative importance of the IP packet and the type of service required for this packet 0 1 2 3 4 5 6 7 Prec. Type of Service 0 Precedence (relative priority) 1000 minimize delay 0100 maximize throughput 0010 maximize reliability 0001 minimize monetary cost 0000 normal service Page 11 ToS evolution definition of ToS Octet changed several times Precedence is used in some networks ToS field is rarely used Today, ToS byte has been replaced by DSCP ToS-based routing Principle Attach a different IGP cost on each link for each ToS value Compute and maintain a forwarding table per ToS value Forward the packets based on their ToS Example d=3, t=10 R2 d=1,t=10 R5 d=10,t=10 R1 d=5, t=10 d=3, t=20 R3 d=1,t=2 R4 d=3,t=5 d=6,t=9 R6 Page 12 d: delay metric t: throughput metric

What happened to ToS routing? Support was added in IS-IS for both CLNP and IP OSPFv2 But ISPs did not clearly request this and no major vendor implemented it... ToS byte was seldom used by applications Main usage was Precedence=6,7 for routing packets ToS routing will be removed from the OSPF and IS-IS specification Page 13 Agenda Routing basics The ToS approach The Integrated Services approach The Traffic Engineering approach Issues with QoS routing in the Internet Page 14

# # # # # " " " " " # # $! Page 15 Integrated services The "hard" approach for QoS Basic hypothesis Some specific applications require QoS delay guarantees bandwidth guarantees Three types of services Guaranteed Service Controlled Load Null QoS will be provided to layer 4 flows Each layer 4 (TCP or UDP) flow will inform the network about its QoS requirements The network will accept or reject the flow based on its requirements and the current state of the network QoS should support unicast and multicast QoS flows should be allowed to coexist with best-effort flows in the same network Integrated services network Provision of integrated services in a network Applications utilize signaling protocol to indicate their QoS requirements QoS Response QoS Request R2 Routers rely on per layer 4 flow scheduling to provide the required QoS Routers perform connection admission control to reserve ressources for each flow R3 Normal IP routing is used to select path towards destination. IP routing is not changed to support Integrated Services R1 R4 R5 R6 Page 16

) ( ( ( ' ' ' & & % An Integrated services router Model of an Integrated Services router RSVP messages Data IP packets RSVP Integrated Services capable router Layer-4 Classification + traffic contract enforcement Admission Control Resource reservation Routing IP packets with reservation IP packets without reservation Routing protocol Data IP Packets Control Forwarding Scheduler Page 17 RSVP RSVP : Resource Reservation Protocol Objectives Support the establishment of unidirectional flows in IP networks different types of flows initially layer 4 flows Suitable for IP unicast should take into account the possibility of route changes without requiring any modification to routing protocols Suitable for IP multicast should fully support the IP multicast service model including dynamic groups Page 18

- -,, + +. * * * RSVP (2) Principles of operation Two important RSVP messages PATH used by sender to inform routers and receivers of the new flow and its required resources RESV no resources are reserved due to reception of PATH used by receiver to actually reserve resources for the flow specified in the PATH message resources are reserved for the IP packets sent by the sender towards the receiver along the path taken by the PATH message RSVP messages are sent inside IP packets Page 19 RSVP : example Unicast flow establishment with RSVP S1 R1 R2 R3 D1 PATH PATH specifies flow and announces amount of traffic to send PATH Routing finds next hop=r2 nothing is reserved PATH PATH D1 knows requirements for this new flow S1 is asked to reserve resources on S1-R1 RESV RESV RESV R3 can reserve enough resources for D1's flow RESV D1 asks R3 to reserve resources on R3-D1 link for this flow Page 20 RESVconf Optional confirmation for D1 will be sent to D1 RSVP messages are sent unreliably as simple datagrams RESV messages must follow same route as PATH messages

2 2 2 1 5 5 5 5 1 1 4 4 0 0 4 4 4 4 3 3 / Constrained routing What should be added to traditional routing algorithms? a way to distribute information about current network state routers must know load of remote links to choose paths meeting constraints for flows with QoS guarantees a way to compute a path subject to constraints current routing algorithms find shortest path how can we find a path with minimum hop count at least 10 Mbps at most 10 msec of delay Page 21 Page 22 Distributing load information Distance vector routing protocols [RIP,BGP] routers conspire to distribute routing table difficult to inform routers of load on remote links difficult to support constrained routing Link state routing protocols [OSPF, IS-IS] routers conspire to distribute network map simple to add information about network load routers distribute link state packets with load info delay is already distributed as the IGP metric bandwidth/link load is main information to distribute tradeoff between frequent distribution (accurate information) and rare distribution (avoid network overload) each router knows topology and load of each link and can find constrained paths

?? @ = < < < < < 8 < < < ; ; ; 7 7 : : : 6 9 R1 Distributing load information (2) Potential problem Link load information is not distributed immediately routers must establish flows based on partial information about the current load in the network D=3 B=6 D=5 B=8 R2 D=3 B=2 R3 2 D=1 B=6 D=1 B=6 1. new flow [B=4] is created between R4 and R6 2. before information about load changes, R3 wants to create a new flow [B=2] towards R6 R5 R4 D=3 B=2 R3 believes that R3-R4-R6 is the best path 1 D=10 B=8 D=6 B=5 R6 Page 23 Page 24 Constraints Three types of constraints on path selection Additive constraint find path minimizing example hop count link delay or cost Multiplicative constraint find path minimizing example loss rate Concave constraint find path containing links whose characteristic is always above a given constraint example bandwidth resource class or color >d 1, d 2,..., d n >d 1, d 2,..., d n

G G F C F F C C E B B D A A Finding a constrained path Single additive or multiplicative constraints apply Dijkstra's algorithm example C=3 R2 C=1 R5 C=10 R1 C=5 C=3 R3 C=1 2 or more additive/ multiplicative constraints unfortunately problem is NP hard need to evaluate all possible paths to find exact solution several heuristics have been proposed in literature to find acceptable solutions R4 C=3 C=6 R6 Page 25 Finding a constrained path (2) Concave constraints fortunately easy to handle remove from the network map all links that do not satisfy the constraint utilize Dijkstra's algorithm on the reduced map example find shortest 3 Mbps from R3 to R6 C=3 B=6 R2 C=1 B=6 R5 C=10 B=8 R1 C=5 B=8 C=3 B=2 R3 C=1 B=1 R4 C=3 B=10 C=6 B=1 R6 utilize only links with some kind of protection Page 26

K K K K K K K J J J N I M M M M H L L L L Page 27 Constrained routing in IP networks Several solutions proposed by researchers Lessons learned Constrained routing should be applied to flows and not on a per packet basis Bandwidth and delay are key constraints delay jitter is less important and difficult to efficiently support Path selection should be performed by the source the source of a flow selects an explicit route at least inside a single area if the path crosses areas of domains, the explicit route will be loos intermediate nodes perform connection admission control but do not perform any constrained routing decision path selection algorithm does not need to be standardized if the new flow is acceptable, establishment continues otherwise the source will have to compute another path Constrained routing and Intserv Page 28 Integrated services was developped at the same time as ATM-Traffic Management Guaranteed Service <-> CBR, rt-vbr Controlled Load <-> VBR, GFR Scalability problems pledged both end-toend ATM and InterServ Constrained routing was used for ATM PNNI was defined and implemented For Integrated services, IETF was relunctant to change anything to IP routing IETF defined is a framework for QoS routing, but no protocol although OSPF extensions were proposed by many researchers and prototypes were implemented

S S R R Q S S S Q Q P P P P P O O S S S S S S A critique of Integrated services Advantages provides per layer 4 flow QoS guarantees GS with delay/ bandwidth guarantees CL with bandwidth guarantees Drawbacks Requires each intermediate router to perform some operations for each layer 4 flow RSVP message processing per layer 4 flow classification classification can become complex for multicast! per layer 4 flow policing/queuing/scheduling a backbone router may see thousands of simultaneous flows! Not all applications are able to express precisely their traffic and QoS requirements Page 29 Scaling issues with integrated services PATH 1. RSVP signaling overhead one RSVP PATH/ RESV per layer 4 flow per refresh period R1 R2 3. Per data packet processing find layer 4 flow state check traffic contract check reservation R3 R4 R5 R6 RESV 2. State information required inside each router Identification of layer 4 flow (IP dest, Protocol, Port) Identification of previous hop (to forward RESV) Reservation state Reserved resources Identification of the senders that benefit from reservations Page 30

U U T T T T T U Agenda Routing basics The ToS approach The Integrated Services approach The Traffic Engineering approach Issues with QoS routing in the Internet Page 31 MPLS Integrating label swapping and IP Pure IP packets Ra Rb Flow 1 Flow 2 LSR1 Flow 3 LSR2 Ingress Edge Router inserts labels on packets sent through backbone Rd Labeled packets LSR3 Core LSR : Label-Switching Router packet forwarding based only on labels Rd Pure IP packets Egress Edge router removes labels before sending packets outside MPLS network Page 32

Label-swapping example Cra Routing table default via BR1 Label table Flow1 : L1 via BR1 Flow2 : L2 via BR1 BR2 Routing table default via BR3 BR1, BR3 : direct Cra, Crb via BR1 Label forwarding table Inport Inlabel Outport Outlabel South L2 S- E L1 BR2 Crd BR3 Routing table default -> Internet BR1, BR2, CRd : direct Cra, Crb via BR1 Flow 1 Flow 2 Cra BR1 BR3 Page 33 Routing table default via BR1 Label table Flow3 : L1 via BR1 Flow 3 Crb BR1 Routing table default via BR3 BR2, Cra, Crb : direct Label forwarding table Inport Inlabel Outport Outlabel West L1 North L2 West L2 East L1 S-W L1 East L2 BR3 Label forwarding table Inport Inlabel Outport Outlabel West L1 East L1 West L2 East L3 N-W L1 North L1 Architecture of a core LSR IP Routing protocol Label Distribution IP Routing table Label table Control Labeled packets Label Forwarding Table Class. Shap. Pol Labeled packets Forwarding Class. Shap. Pol Labeled packets Page 34

X W W W V Architecture of an ingress edge LSR IP Routing protocol Label Distribution IP Routing table Label table Control IP packets Identify FEC Class. Shap. Pol Labeled packets Forwarding Attach Label Class. Shap. Pol Labeled packets Page 35 The Traffic Engineering problem Problem Shortest path chosen by IP routing does not always lead to a good network utilization fish problem R1 many packets towards R9 R4 R8 many packets towards R8 R2 R3 R5 Shortest path tree rooted on R3 How to better optimize the network utilization? How to react to changes in traffic conditions R6 R7 R9 Page 36

` ` ` \ \ _ [ [ [ [ [ ^ ^ ^ Z Z Z Z ] Y Y Page 37 Principle MPLS-based traffic engineering In each POP, routers establish MPLS tunnels with bandwidth reservations towards remote routers MPLS's label swapping will ensure that packets will follow chosen path full mesh of MPLS tunnels Issues How to select the router-router path? Back to constrained routing with some adaptations How to establish the MPLS tunnel Slightly change RSVP to distribute MPLS labels path is computed by the source and specified as an explicit route in the RSVP Path message Labels are allocated by the RESV messages How to change bandwidth or route a LSP? Make-before-break in RSVP-TE Constrained routing and MPLS-based traffic engineering What should we add to OSPF for MPLS-TE? Bandwidth information for each link OSPF-TE will attach unreserved bandwidth information to each link one amount of unreserved bandwidth for each of the 8 preemption levels preemption levels are used to indicate importance of LSPs Maximum bandwidth of each link This is the physically available bandwidth on the link Maximum reservable bandwidth Maximum amount of bandwidth that can be reserved by MPLS tunnels Could be higher than, equal to or lower than maximum bandwidth Page 38

d d d d d c c b b e e e e e a Constrained routing and MPLS-based traffic engineering (2) What should we add to OSPF for MPLS-TE? Traffic engineering metric Could be different from the normal IGP metric which usually is set based on rerouting and bandwidth requirements Common usage of TE metric is to indicate delay to establish tunnels supporting VoIP or tunnels carrying EF traffic We're back to ToS routing, but on a per-tunnel basis! Resource class or color Allows to identify all the links with the same characteristic, essential to establish protection LSPs all satellite links are in blue all links provided by FT are in red all links provided by Sprint are in yellow Page 39 Agenda Routing basics The ToS approach The Integrated Services approach The Traffic Engineering approach Issues with QoS routing in the Internet Page 40

i i i l h h l l h h h h k g g g k k k k j j f Page 41 Dynamics of QoS routing with MPLS-TE How stable is an MPLS-TE network when link failures occur Sprint shows that failures are common events each failure may cause the rerouting of some MPLS tunnels which will change the bandwidth available dampening mechanisms on LSP generation threshold-based LSP generation holding time on LSP generation Routers are upgraded Most ISPs have 2 hours maintenance windown during the night many MPLS tunnels will need to be rerouted Change in MPLS tunnels bandwidth reservations some vendors propose to measure per tunnel bandwidth and adapt reservation on hourly basis some vendors propose to reoptimize MPLS-tunnel paths on a regular basis to preserve network optimality Page 42 How to extend QoS routing across areas? IGPs used by large ISPs ISIS : mainly single area deployments OSPF : often multi-area deployments How to establish interarea MPLS tunnels? interarea IP routing is not as clean as PNNI Tunnel headend does not know all topology to select best path to reach destination What kind of TE information should be leaked across areas? Possible solutions include headend selects loose route and area border router updates the route headend contacts path computation server to select the route, based on full area knowledge

p p p o o n n n n n m m How to extend QoS routing across domains? Several large ISPs want to use MPLS tunnels between ASes large company with several ASes research networks (GEANT, I2,...) What is required to establish those tunnels? RSVP-TE allows to specify interdomain routes But BGP is a path-vector protocol that does not provide any TE information no metric no bandwidth attribute no topological information Headend should define loose route that will be improved by AS border routers optimality of interdomain paths is unknown Page 43