L4: ISPs, Backbones and Peering 15 September 2015 (From Kobe) Bill Nace Material from Computer Networking: A Top Down Approach, 5 th edition. J.F. Kurose and K.W. Ross
Administrivia Norton2010 Paper Review for today Lab0 should be posted soon Due 24 Sep 2
Last Lecture Internet TCP/IP architecture Layering not strictly enforced Hourglass -- IP ueber alles Allows rapid innovation at layers below Flexible applications / services above IETF process Rough consensus and running code End-to-end argument Does it still apply? 3
traceroute ISPs and Backbones Peering and Settlements Peering Evolution Interconnections 4
Backbones In the beginning... of Internet time Single backbone: NSFNet Everyone on the Internet was on NSFNet A backbone network enables all connected end-hosts (users and companies) to communicate with each other No interconnection problems 5
Backbones (2) Commercial backbone providers emerged Technology transfer! A Good Thing If US Government was managing the Internet, might not be so successful We all expect universal Internet connectivity 6
A Packet s Journey Packets travel across many networks Particular protocols will be studied later This lecture motivates why routing mechanics are necessary CMU Pgh A B G PennRen Little ISP C H Some NW Huge ISP D Tier 1 ISP F Cheap ISP Tier 1 ISP Tier 1 ISP Some ISP 14-740: Fall 2014 E J CMU Kobe 7
CMU s Infrastructure Two 10 Gigabit Ethernet links (singlemode fiber) to Pittsburgh Supercomputer Center From Cisco 6500s located in Cyert and Stever House Redundancy links to PennRen / Level3 over 1Gbps rate-limited 10 Gigabit Ethernet PennRen 1Gbps/10Gbps External Internal PSC Internet2 10Gbps 10Gbps Level3 Comodity Internet 1Gbps Money $40K per year to PennRen A PA state education network $100K to PSC, $100K to Level3 $300K per year on fiber leases CMU Internal
Interfaces: Transit Transit / provider-customer ISP sells access to another ISP or company e.g. GE buys transit (or access to Internet) from Verizon 9
Interfaces: Peering Reciprocal access to each other s customers Usually free exchange of traffic DO NOT serve as transit for 3rd party data E.g. Google and MSN peer with each other, so email messages between Gmail and Hotmail are transferred directly, without going through their transit providers These relationships are confidential business secrets Roughly hierarchical, though the topology is flattening 10
Tier 1 ISP Internet backbone providers Peers with every other Tier-1 in Internet Region Who is a major ISP in your country? US? Japan? India? Careful: Tier-1 is an overloaded and misleading term 14-740: Fall 2014 11
Images courtesy of Level-3
>4200 networks in 142 nations, many at OC-768 speeds Images courtesy of Verizon
Tata: "Discover the World's Largest Global Footprint" Image courtesy of tatacommunications.com/network
Typical Infrastructure High speed links Level3 operates 10Gbps (OC-192) IP backbone Installing 40Gbps DWDM networks High performance routers Over-provisioned bandwidth 40 ms delay within region <1% packet loss Global presence (or at least multi-continent) 15
Typical Tier-1 Relationships Directly connected to other Tier-1 ISPs (i.e. peer with) Connected to a large number of Tier-2 ISPs Vertically integrated: sell services directly to customers International in coverage Ecosystem: do not buy transit from another provider in order to reach the whole Internet 16
Why do Tier-1 ISPs need to peer with each other? No single Tier-1 ISP can reach the whole Internet on its own Internet is a network of networks But Tier-1 ISPs have a restrictive peering policy Do not peer with other non-tier-1 ISPs They are potential revenue generating customers No incentive to accept additional peers 14-740: Fall 2014 17
What is a Tier-2 ISP? Network infrastructure is usually regional Customer of Tier-1 ISP(s) needs to buy transit Provider of customers also re-sells this transit Peers with other Tier-2 ISPs settlement free Tier-2 ISP pays Tier-1 for connectivity to rest of internet Multi-connection relationships possible Tier-2 ISP Tier 1 ISP Tier 1 ISP IXP Tier-2 ISP Tier 1 ISP Tier-2 ISP Tier-2 ISP Tier-2 ISPs also peer privately. Interconnection can be direct or via IXP 18
Thinking time Why do Tier-2 ISPs need to buy transit from a Tier-1 provider? 14-740: Fall 2014 19
Thinking time Why do Tier-2 ISPs peer with each other? 14-740: Fall 2014 20
Open Peering Policy Tier-2 ISPs have an open peering policy Peer with anyone possible Costs of peering have to be balanced against gains for a Tier-2 ISP Management cost: usually settlement-free peering means both parties should send approx equal amounts of traffic to each other Maintenance cost: extra equipment, transmission capacity to meeting point, exchange and other fees 21
Content Providers Do not sell transit Category A: Focus on content creation Do not want to operate a network Do not have expertise in maintaining peering relationships router, policy, negotiations, No Peering Policy Category B: Sophisticated, large-scale players Use peering to improve user experience Open Peering Policy 22
Peering Ecosystem Internet Region Full Mesh Peering Tier 1 ISPs Transit Free Partial Mesh Peering Tier 2 ISPs Must Buy Transit Generally No Peering Content / Enterprise Companies Must Buy Transit Things are never so clear-cut...
traceroute ISPs and Backbones Peering and Settlements Peering Evolution Interconnections 24
US Evolution 1999/2000 economic collapse Telecom sector Need to rethink their business model Need to cut costs General dotcom bust Lots of cheap equipment on ebay Transit rates drop Upstream provider @Home for cable companies went bankrupt Peer-to-peer file sharing grows exponentially in popularity 25
Transit Prices are Falling Price per Mbs 63% per year decline Luckily, volume is increasing even more Somewhat bad for Tier-1 ISPs 26
Cable Companies are Peering The P2P effect Residential cable modem users About 40%-75% of traffic is P2P and destined for other cable companies or other access heavy ISPs Peering allows cable companies to offload this traffic from transit connections Good for cable companies (save money) Bad for Tier-1 ISPs 27
Content Companies are Peering Network savvy, large scale enterprises MSN, Yahoo, Google, Ebay, Walmart, Yahoo has an open peering policy Reduce transit costs Improve end-user experience Good for content companies Bad for Tier-1 ISPs 28
Cable Companies Peer with Content Companies Send traffic directly from content providers to cable modem users Reduce transit costs Bad for Tier-1 ISPs Improve performance Can you predict another major disruption in the Peering Ecosystem? Tier 2 ISPs Tier-1 ISPs Cable Companies Content Producers LSNSC
traceroute ISPs and Backbones Peering and Settlements Peering Evolution Interconnections 30
Interconnection How do two networks interconnect with each other? We are talking about routers in two different companies, under separate administrative control 31
Public Peering U.S. government decided to let commercial companies take over management of the backbone networks 1991: Commercial Internet Exchange (CIX) 1 router in Santa Clara, CA 1995: Network Access Points (NAPs) in SF, Chicago, NY, D.C. More recent: Internet Exchange Point (IXP) 30 in US, 6 in Japan More is good because of congestion, at one point 20% of traffic going through the 1st NAP was dropped! 32
ISP Connections Tier-1 providers interconnect privately (Peer) Tier 1 ISP Tier 1 ISP Tier 1 ISP IXP (Meet-me room) Tier 1 ISP Tier-1 providers also interconnect at Internet Exchange Points (IXP) (30 in US)
Private Peering Two networks bypass IXPs and directly peer In response to congestion at some IXPs Business reasons Also more cost-effective There are limited number of IXPs at major regions Can privately peer in other locations Sometimes take a hybrid approach Public exchange for some peers Private meeting point for others 34
Provider-customer Interconnection Point-of-presence (PoP) Customer typically leases a high-speed link from a 3rd party telecomm provider Directly connects a router at a provider s PoP 35
Various Methods How do two networks interconnect with each other? Many methods: Public Peering, Private Peering, Provider-customer PoP There are really no industry-specific regulations, mostly purely driven by commercial and economic forces 36
Summary Hierarchy of ISPs (Tier-1, Tier-2, Content Provider) Peering lets ISPs avoid paying for transit Public peering happens at IPX Private peering also exists 37