Internet2 Network Operations Update Chris Robb Internet2 Manager, Network Operations 28 April 2009 - Arlington Spring Members Meeting 1
Infrastructure Portfolio 2
Colocation Facilities Optical Network Redundant OADM Optical Regeneration IP Network Ciena Network 3
Internet2 Combined Infrastructure Topology Portfolio of network infrastructure and services across the Internet2 footprint Seattle Portland Rieth Syracuse Boise Tionesta Reno Eureka Buffalo Ogden Sacramento Rawlins Sunnyvale Salt Lake City Phoenix IP node Ciena node Redundant OADM node Colocation node Optical regeneration node Kansas City Philadelphia Indianapolis Washington DC Denver Louisville Raleigh Tucson Nashville Tulsa Albuquerque Rancho de la Fe El Paso Charlotte Atlanta Birmingham Dallas Houston Valentine Sanderson New York (2) Pittsburgh Raton Los Angeles San Diego Cleveland Omaha St. Louis San Luis Obispo Boston Detroit Chicago (2) Heartwell San Francisco Albany Jacksonville Mobile San Antonio Tallahassee Baton Rouge 4
Colocation Facilities 30 Facilities across the US 27x Level(3) 1x Pacific Northwest Gigapop, Seattle 1x 32 Avenue of the Americas, New York 1x 710 North Lakeshore Drive, Chicago Dedicated suites or colocation areas Hosts Internet2, ESNet and some Connector equipment 5
Optical Network Infinera optical transmission equipment Managed by Level(3) Communications Internet2 has direct provisioning access to create/edit/delete circuits Internet2 has direct TL-1 access to monitor the network Three types of Infinera nodes Redundant Add Drop (28) 109,500 Regeneration (28) 92,000 Optical Amplification (193) 113,269 120000 90000 60000 Total Lit Route Kilometers: April 2009 October 2009 April 2009 30000 0 6
Optical Network Operational Reationships Level(3) Broomfield Level(3) Atlanta Atlanta-Indianapolis DS-3 to Infinera Management Denver-Sunnyvale DS-3 to Infinera Management Weekly Calls 7
IP Network Internet2 IP Network Topology Internet2 IP router core topology and the backbone links that interconnect them Seattle Chicago T1600 Salt Lake City New York Kansas City Washington DC Los Angeles 10 Gbps 20 Gbps Atlanta Houston 8
IP Network Juniper Networks Router Platform 8x 1x T1600 10-20G Backbone Weekly staff evaluation of backbone utilization Internet2 IP Network Topology Internet2 IP router core topology and the backbone links that interconnect them Seattle Chicago T1600 Salt Lake City New York Kansas City Washington DC Los Angeles 10 Gbps 20 Gbps Atlanta Houston Typical Internet2 IP Core Node: 9
Internet2 Offered Load 7 6.5 6 5.5 Cisco GSR Qwest Footprint 2.5G Backbone Juniper Qwest Footprint 10G Backbone Juniper Level(3) Footprint 10G+ Backbone 6.7 Petabytes # Petabytes Transfered/Month 5 4.5 4 3.5 3 2.5 2 2 Petabytes 2.5 Petabytes 4.1 Petabytes 2.3 Petabytes 1.5 1 1.2 Petabytes 0.5 2 Terabytes 0 6/1/99 3/1/00 12/1/00 9/1/01 6/1/02 3/1/03 12/1/03 9/1/04 6/1/05 3/1/06 12/1/06 9/1/07 6/1/08 3/1/09 Time R/E Traffic R/E + CPS Traffic 10
7 6.5 6 5.5 Cisco GSR Qwest Footprint 2.5G Backbone Juniper Qwest Footprint 10G Backbone Juniper Level(3) Footprint 10G+ Backbone 6.7 Petabytes Internet2 Offered Load # Petabytes Transfered/Month 5 4.5 4 3.5 3 2.5 2 2 Petabytes 2.5 Petabytes 4.1 Petabytes 2.3 Petabytes 1.5 1 1.2 Petabytes 0.5 2 Terabytes 0 6/1/99 3/1/00 12/1/00 9/1/01 6/1/02 3/1/03 12/1/03 9/1/04 6/1/05 3/1/06 12/1/06 9/1/07 6/1/08 3/1/09 Time Measurement Methodology Data compiled from 5 different monitoring systems over 10 years Measurement performed at network edge (PE) interfaces 3 Primary Phases 1999-2002 OC-48 Backbone 2003-2007 OC-192 Backbone 2007-Present Multiple 10G Backbone Analysis R/E traffic loads have been stable over the last 5 years Large amount of CPS traffic in past 2 years 11
Average Interface Speed 9000 8.3Gbps 8000 7000 6000 Mbps 5000 4000 Cisco GSR Qwest Footprint 2.5G Backbone Juniper Qwest Footprint 10G Backbone 3.4Gbps 4.8Gbps Juniper Level(3) Footprint 10G+ Backbone 3000 2000 1000 1.2Mbps 465Mbps 0 1 Jan-02 Jul-02 Jan-03 Jul-03 Jan-04 Jul-04 Jan-05 Jul-05 Jan-06 Jul-06 Jan-07 Jul-07 Jan-08 Jul-08 Time 12
Internet2 Ciena Network Internet2 Ciena Network Topology Internet2 Ciena network topology and the backbone links that interconnect them Seattle Boston Chicago Sunnyvale Salt Lake City Kansas City Cleveland New York Pittsburgh Philadelphia Indianapolis Washington DC Denver Louisville Los Angeles Nashville Albuquerque Charlotte Atlanta Houston Jacksonville Baton Rouge 13
Internet2 Ciena Network Internet2 Ciena Network Topology Internet2 Ciena network topology and the backbone links that interconnect them 22 Ciena nodes 21 half-bay CoreDirector CI chassis 1 full-bay CoreDirector chassis (Chicago) Serves three primary functions Backhaul of sub-10g signals to IP router sites Sub 10G static circuit services Infrastructure for Internet2 Dynamic Circuit Seattle Boston Chicago Sunnyvale Salt Lake City Kansas City Cleveland New York Pittsburgh Philadelphia Indianapolis Washington DC Denver Louisville Los Angeles Nashville Albuquerque Charlotte Atlanta Houston Jacksonville Baton Rouge Network OC-192 core backbone Ethernet capabilities at the edge Small number of SONET client signals for legacy connections Ethernet frames mapped to timeslots on SONET backbone Typical Internet2 Ciena Core Node: 14
Operational Statistics 15
NOC Trouble Ticket Statistics The charts to the right show Internet2 NOC Trouble Ticket breakdowns over the prior 6 months 1,216 Tickets across Optical, IP and CPS Networks 16
NOC Trouble Ticket Statistics Internet2 IP Network Observations 854 Tickets Majority of tickets are questions about the network Unscheduled Outages are significant and require followup 17
NOC Trouble Ticket Statistics Internet2 CPS Network Observations 217 Tickets Most NOC activities revolved around outages and maintenances Not many questions from CPS peers. Most are from CPS Connectors 18
NOC Trouble Ticket Statistics Internet2 Optical Network Observations 145 Tickets Most NOC activities revolved around outages and maintenances Not many questions from CPS peers. Most are from CPS Connectors 19
Optical Network Availability 100.000 99.875 99.750 99.625 9/8/08 10/6/08 11/3/08 12/1/08 12/29/08 1/26/09 2/23/09 3/23/09 99.500 Aggregate Weekly Availability over all 62 DWDM Segments (scheduled and unscheduled) Maximum: 100% Minimum: 99.90% (utility cut near Niles, MI) Average: 99.989% 20
Optical Outage Case Study (something recent) 21
Sunday, April 26-91 Degrees and Sunny 12:51 PM ET - Internet2 observes Loss of Signal (LOS) alarms on Philadelphia and Washington DC Infinera nodes 5:42 PM ET - Level(3) identifies the location and has splice crew on-site 5:55 PM ET - Internet2 begins re-route of three key circuits via Cleveland 2x Internet2 IP Network 1x ESNet Access Circuit 6:00 PM ET - Circuit re-route complete 12:10AM ET - Splice crews complete repair 2:53AM ET - Three re-routed circuit normalization begins 2:56AM ET - Three re-routed circuit normalization completes 22
Latency of New York- Washington DC Traffic via Chicago Router via Cleveland optical path Original Path circuit restoration 23
Questions? chrobb@internet2.edu Twitter: chrobb 24