This document provides guidelines on scaling the performance of DS3 interface (NM-1T3/E3) for the Cisco 2811/2821/2851/3825/3845 Integrated Services Routers. The analysis provides following test results; Performance Throughput for IMIX traffic Performance Throughput for 64 Bytes traffic Test Methodology: To be certain of accurate results, the DS3 throughput tests were configured with a frame rate at which the ISR does not lose packets (No Drop Rate). This test is performed in order to know the ISR s maximum throughput rate. Resulting frame rate from this throughput test is then sustained for an adequately longer period to obtain a stabilized CPU utilization value for the traffic with the characteristic services configured if any. An additional data point is obtained further by backing the traffic down, so as to obtain the throughput for each platform at a CPU utilization of 65%. This test is repeated for 64 Bytes and IMIX traffic with and without relevant services configured on the device under test. These tests were performed sending traffic across On-board FE (or) GE to NM-1T3/E3 interface as shown below. Cisco ISR LAN to LAN Connectivity Traffic Type 2811 FE to NM-1T3/E3 IP 2821 Gigabit Ethernet (GE) to NM-1T3/E3 IP 2851 GE to NM-1T3/E3 IP 3825 GE to NM-1T3/E3 IP 3845 GE to Nm-1T3/E3 IP Table-1 *All numbers represent a unidirectional throughput Packet Sizes for IP Traffic: 64 Bytes and Internet Mix (IMIX) IMIX Traffic is defined as the following streams: 7 data streams of 64 byte packets 4 data streams of 570 byte packets 1 data stream of 1518 byte packets Actual traffic pattern is [64, 64,570,64,64,570,64,1518,570,64,64,570] The average packet size computes to 354 bytes. [1518 + (7*64) + (4*570)/12]=354 DS3 Scalability Test-Bed setup: Figure-1 Report by Srinivas K, TME ATG Page 1 of 8 September 2007
Test results obtained from the tests are based on Cisco IOS Release 12.4.9T3/T5 for all the ISR Series. The services tested while scaling DS3 performance are Quality of Service, Access Control Lists (ACL) and IPSec VPN. The access lists are always matched to the last ACE in the list. QoS service configured consists Classification and Queuing mechanisms IPSec is configured for ESP-3DES ESP-SHA-HMAC transform set, pre-shared keys and 3-DES encryption. Services are added one by one and corresponding CPU utilization is recorded. Binary search methodology is followed to arrive on the NDR rate for the DUT. Then step rate is used and sustained for adequate amount of time to record accurate CPU utilization. Results: Throughput* & CPU table: 64 Bytes data traffic: Platforms / Services No Services ACL QoS+ACL QoS+ACL +IPSec / Throughput Mbps Mbps Mbps Mbps No Services CPU Utilization (%) ACL QoS+ ACL QoS+ACL+ IPSec 2811 41.00 10.44 6.30 2.45 99 98 99 99 2821 43.38 22.47 13.40 8.69 99 99 99 99 2851 43.57 28.2 21.5 11.41 99 99 99 99 3825 43.74 35.70 31.73 15.70 82 99 99 99 3845 43.6 43.53 39 22.15 86 99 99 99 Throughput* for CPU (65%) table: 64 Bytes data traffic: Platforms / Services No Services ACL QoS+ACL QoS+ACL+IPSec / Throughput Mbps Mbps Mbps Mbps 2811 8.2 3.45 -na- -na- 2821 14.51 7.58 -na- -na- 2851 17.66 9.464 -na- -na- 3825 33.44 21.45 -na- -na- 3845 30.96 24.28 -na- -na- Throughput* & CPU table: IMIX traffic: Platforms / Services No Services QoS QoS+ACL QoS+ACL +IPSec / Throughput Mbps Mbps Mbps Mbps No Services CPU Utilization (%) QoS QoS+ ACL QoS+ACL+ IPSec 2811 44.04 44.40 43.50 10.66 38 98 99 99 2821 44.40 44.45 44.45 33.62 28 61 78 99 2851 44 44 44 40.607 28 51 67 98 3825 44.15 44.50 44.50 43.80 16 30 45 95 3845 44.50 44.50 44.50 44.42 16 26 39 87 Report by Srinivas K, TME ATG Page 2 of 8 September 2007
Throughput* for CPU (</= 65%) table: IMIX data traffic: Platforms / Services No Services QoS QoS+ACL QoS+ACL+IPSec / Throughput Mbps Mbps Mbps Mbps 2811 44 -na- -na- -na- 2821 44.40 44.45 na na 2851 44 44 44 na 3825 44.50 44.50 44.50 26.90 3845 44.50 44.50 44.50 33.40 Summary on the analysis: 1. Test traffic is sent between On-board Ethernet interface and the Serial interface (LAN to WAN). Throughput and performance may vary depending on the interface types (ex: FE/GE or ATM). The NM-1T3/E3 card is a dual controller card configured for T3 speed. 2. Additional IOS services such as Firewall, IPS and NAT will add more CPU overhead and latency bringing the performance further down. 3. The services configured are QoS, ACL and IPSec VPN. In QoS, classification and queuing functions are considered for tests. The queuing configured comprises of LLQ, CBWFQ and default queue (FIFO). The classification method used involves matching criteria with IP Precedence. 4. The ACL configured has 20 ACEs. This number has been arrived after considering the average length of ACL in a real time environment, where a packet would find a matching entry. 5. The oversubscription of link is done by pumping Ethernet traffic slightly more than T3 link bandwidth. This rate is kept constant for all the platforms except for the lower 2800s which cannot fill this line rate before utilizing all of its CPU cycles. 6. Throughput data for 65% CPU is tested for both IMIX and 64 Bytes streams. Though for 64 Bytes, it has not been possible to obtain results with QoS services. This is because of a spike in CPU utilization which would be induced at the moment Software Queuing becomes active by the eventual over subscription of the link. 7. Unlike usual performance listings, this report represents uni-directional numbers only instead of an aggregate traffic. 8. A recommended data rate for each platform for three different deployments is given below. Each table has recommended throughput rate with ACL, QoS and IPSec services configured. All values are arrived on optimum CPU utilization value of 65% or below. Report by Srinivas K, TME ATG Page 3 of 8 September 2007
Projected throughput recommendations for DS3 link on Integrated Service Routers Routing Platforms Direct Internet Access WAN Edge (with QoS and ACL) WAN Edge (QoS, ACL and Crypto) WAN Edge with Crypto (on Small Packets) Cisco 2811 35 Mbps Not recommended Not recommended Not Recommended Cisco 2821 DS3 Line rate 15 Mbps Not recommended Not Recommended Cisco 2851 DS3 Line rate 20 mbps Not recommended Not Recommended Cisco 3825 DS3 Line rate 30 Mbps 25 Mbps Not Recommended Cisco 3845 DS3 Line rate DS3 Line Rate 32 Mbps Not Recommended 9. Cisco 3845 has more powerful CPU than Cisco 3825. The differences in performances between them will be much more evident as more services are employed on the router. Positioning also considers CPU power. 10. Recommendations reveal that ISRs are not ideal to be deployed under WAN Edge solutions with QoS, ACL and Crypto configurations on it, when the traversing traffic constitutes only small packets of 64 Bytes. 11. Similarly, deployment considerations can be made only for 3800 series platforms for WAN Edge solutions with QoS, ACL and Crypto services configured if the traversing traffic is an Internet Mix (IMIX)* (See the IMIX definitions in the first page). However, the maximum data throughput that can be obtained in such conditions are within the rates mentioned in the table above for each of those platforms. (See section WAN Edge (QoS, ACL and Crypto)). 12. The 2811 can sustain the above mentioned line rate with IMIX for few minutes with the corresponding CPU utilization, after which the CPU tends to shoot to almost double the utilization. Alongside there is fractional packet loss after this delay which hints the hardware buffer overflow. Recommended QoS Considerations Get to the basics first and follow the three foremost steps of Identify, Quantify and Prioritize the traffic. Identification involves assessing the mission-critical and latency prone applications. Use probes (like RMON) and get as much information on applications as possible which are traversing across the WAN link. Then the bandwidth required for each of them has be assessed (Use sniffers and similar software to assess the pps, packet generation, delay, sensitiveness etc). Now write policies to prioritize the traffic. 1. Classification and marking considerations: There can be potential QoS bottleneck in classifying the traffic: Mark one of these IP QoS marking fields-precedence and DSCP- to maximize the benefits of reducing classification overhead by the other QoS tools enabled in the network. Also, because they are part of the IP header, are the only fields that can be marked and carried from end to end in the network. Classify and mark as close to the ingress edge as possible. The number of the classes defined in QoS configs can be detrimental if they exist is large numbers. Less class numbers the better. Report by Srinivas K, TME ATG Page 4 of 8 September 2007
Many applications can be considered mission-critical. However, if too many applications are classified as missioncritical, they will contend among themselves for bandwidth, with the result of dampening QoS effectiveness. To the extreme, a regular FIFO link (no QoS) is scheduled in the same manner as a link where every application is provisioned as mission-critical. General recommendation of classification is to restrict not more than three applications as mission critical ones. Note: - Matching to IP access lists is more processor-intensive than matching based on other criteria. Note: - The sequence of the class: Place the most commonly used matching criteria in the beginning. It will help improve the classification process. 2. Interactive Video Interactive Video or IP Video Conferencing (also called as IP/VC) is recommended to be marked AF41. A downward marking can be done in case of dual-rate policing though. Do overprovision for the LLQ by twenty percent (20%) of the IP/VC rate. This will take into account the IP/UDP/RTP headers and the Layer2 overhead. Cisco IOS may include a 200ms burst size which may be just sufficient for low speed links (a couple of T1s) and high speed links need higher numbers. There is no clear cut formula for predicting the burst size parameters for the IP/VC streams in cases were they are continually added. The point to remember here is that the default burst size parameter for the LLQ should require a tuning as the IP/VC streams are added. And this is likely to be a trial and error method. Note: - WRED is more effective on TCP-based flows than UDP-based flows, such as interactive video. 3. Rate Limiting CAR (Committed Access Rate) embodies a rate limiting feature of policing the traffic in addition to its packet classification feature. CAR propagates bursts. It does no smoothing or shaping of traffic, and therefore does no buffering and adds no delay. It is highly optimized to run on high-speed links DS3. CAR rate limits may be implemented either on input or output interfaces or sub interfaces including Frame Relay and ATM sub interfaces. 4. Comparing CAR and Class-Based Policing Cisco recommends using the modular QoS CLI features when possible to implement quality of service in your network. Use class-based policing through the police command in a service policy to implement rate limiting without buffering or queuing. Avoid using CAR, for which no new features or functionality is planned. Cisco will continue to support CAR for existing implementations using this method. Note: - There are three actions for Class based Policer (Confirm, Exceed and Violate) while only two for CAR (Confirm and Exceed). Cisco Router Configuration: 1 IOS Configuration with No Services DS3-2821# hostname DS3-2821 card type t3 1 Report by Srinivas K, TME ATG Page 5 of 8 September 2007
no aaa new-model ip cef controller T3 1/0 interface GigabitEthernet0/0 ip address 60.60.60.1 255.255.255.0 duplex full speed 100 interface Serial1/0 ip address 10.10.10.1 255.255.255.0 encapsulation ppp dsu bandwidth 44210 max-reserved-bandwidth 100 ip route 0.0.0.0 0.0.0.0 10.10.10.2 2 IOS Configuration with QoS and ACL services card type t3 1 resource policy ip cef controller T3 1/0 class-map match-all PREC-3 match ip precedence 3 class-map match-all PREC-5 match ip precedence 5 policy-map HQOS class PREC-5 priority 13664 20000 (priority class with assured bandwidth) (LLQ) class PREC-3 (CBWFQ) bandwidth 1168 class class-default (FIFO) bandwidth 1142 interface GigabitEthernet0/0 ip address 60.60.60.1 255.255.255.0 duplex full speed 100 interface Serial1/0 (DS3 interface) ip address 10.10.10.1 255.255.255.0 ip access-group 101 out encapsulation ppp dsu bandwidth 44210 max-reserved-bandwidth 100 Report by Srinivas K, TME ATG Page 6 of 8 September 2007
service-policy output HQOS ip route 0.0.0.0 0.0.0.0 10.10.10.2 access-list 101 deny tcp any any eq 1001 access-list 101 deny udp any any eq 1002 access-list 101 deny tcp any any eq 1003 access-list 101 deny udp any any eq 1004 access-list 101 deny tcp any any eq 1005 access-list 101 deny udp any any eq 1006 access-list 101 deny tcp any any eq 1007 access-list 101 deny udp any any eq 1008 access-list 101 deny tcp any any eq 1009 access-list 101 deny udp any any eq 1010 access-list 101 deny tcp any any eq 1011 access-list 101 deny udp any any eq 1012 access-list 101 deny tcp any any eq 1013 access-list 101 deny udp any any eq 1014 access-list 101 deny tcp any any eq 1015 access-list 101 deny udp any any eq 1016 access-list 101 deny tcp any any eq 1017 access-list 101 deny udp any any eq 1018 access-list 101 deny tcp any any eq 1019 access-list 101 permit ip host 60.60.60.3 any (Last Entry Matching) DS3-2821# 3 IOS Configuration with Crypto card type t3 3 no aaa new-model resource policy ip cef voice-card 0 no dspfarm controller T3 3/0 crypto isakmp policy 1 encr 3des authentication pre-share crypto isakmp key 12345 address 10.10.10.2 crypto ipsec transform-set TS esp-3des esp-sha-hmac crypto map ipsectest 1 ipsec-isakmp description #crypto map across DS3 Link# set peer 10.10.10.2 set transform-set TS match address 101 Report by Srinivas K, TME ATG Page 7 of 8 September 2007
interface GigabitEthernet0/0 ip address 60.60.60.1 255.255.255.0 duplex full speed 100 media-type rj45 interface Serial3/0 (DS3 interface) ip address 10.10.10.1 255.255.255.0 encapsulation ppp dsu bandwidth 44210 crypto map ipsectest max-reserved-bandwidth 100 ip route 0.0.0.0 0.0.0.0 10.10.10.2 access-list 101 permit ip 60.60.60.0 0.0.0.255 70.70.70.0 0.0.0.255 control-plane DS3 Circuit: - A brief overview Digital Signal (DS) is a system of classifying digital circuits according to the rate and format of the signal (DS) and the equipment providing the signals (T). DS and T designations have come to be used synonymously so that DS1 implies T1, and DS3 implies T3. A DS3 line (also known as a T3 line) is a high-speed connection capable of transmitting data at rates up to 45 Mbps. One DS3 line is equal to approximately 672 regular voice-grade telephone lines and it is fast enough to transmit realtime video and large databases over a network. The DS3 signal itself is composed of 28 DS1 signals and is constructed using a two-step multiplexing process. First, the 28 DS1 signals are multiplexed into seven DS2 signals. Second, the seven DS2 signals are multiplexed into one DS3 signal. Each multiplexing step uses bit stuffing to handle the different input frequencies. Overhead bits provide alignment, error checking, in-band communications, and bit stuffing control information. DS3 Specifications: Line rate: 44,736,000 b/s Signals: 7 DS2 signals = 28 DS1 signals Overhead bits: 56 bits total/frame F-bits (framing) 28 bits/ M-bits (multiframing) 3 bits/ C-bits (stuffing) 21 bits X-bits (message) 2 bits/ P-bits (parity) 2 bits/ Data bits between overhead bits 84 DS3 service can be deployed for a wide verity of applications. Common deployments of it include DS3 point-to-point, DS3 internet, DS3 frame relay, DS3 voice and DS3 VPN. The pricing for these connections varies depending on the carrier, location of service and the application for which the connection is being used. References: - http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122cgcr/fqos_c/fqcprt4/qcfpolsh.htm http://www.cisco.com/warp/public/105/policevsshape.html http://www.cisco.com/en/us/tech/tk543/tsd_technology_support_category_home.html Report by Srinivas K, TME ATG Page 8 of 8 September 2007