1 Global Traffic Exchange among Internet Service Providers (ISPs) OECD - Internet Traffic Exchange Berlin, June 7, 2001 J. Scott Marcus, Chief Technology Officer (CTO)
2 Economics of Internet Interconnection Background Internet interconnection in North America International Internet interconnection Economic modeling of Internet backbone peering The off-net pricing principle Implications for international interconnection Scaling challenges to the peering system Cross-provider differentiated services, measurements and SLAs
3 Internet Interconnection in North America Historical roots Peering and transit Shared and direct peering Shortest exit Hierarchical structure
5 Privatization of the Internet (NSF 93-52) ISP ISP ISP Backbone ISP ISP ISP NAP Backbone ISP Chicago NAP NAP New York ISP San Francisco Los Angeles Backbone ISP Washington DC NAP ISP Atlanta Dallas Backbone ISP Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.
6 Peering and Transit Peering is usually a bilateral business and technical arrangement, where two providers agree to accept traffic from one another, and from one another s customers (and thus from their customers customers). Peering does not include the obligation to carry traffic to third parties. Transit is usually a bilateral business and technical arrangement, where one provider (the transit provider) agrees to carry traffic to third parties on behalf of another provider or an end user (the customer). In most cases, the transit provider carries traffic to and from its other customers, and to and from every destination on the Internet, as part of the transit arrangement. Peering thus offers a provider access only to a single provider s customers; transit, by contrast, usually provides access at a defined price to the entire Internet. Historically, peering has often been done on a bill-and-keep basis, without cash payments, where both parties perceive roughly equal exchange of value; however, there is often an element of barter.
7 Shared and Direct Peering A few shared global traffic exchange points. Smaller domestic shared traffic exchange points for regional concentration and exchange of traffic. Direct traffic exchange carries most Internet backbone traffic. Even though shared traffic exchange points are losing market share, their traffic is likely to continue to grow in absolute terms. Carrier hotels and fiber interconnects - an emerging trend that seeks to provide the best of both worlds. Whether shared or direct, the prevailing pattern is shortest exit routing - the sending provider hands off traffic at the point most convenient to the sender.
8 Shared and Direct Peering Backbone ISP Router Direct Interconnection Router Router Backbone ISP Router Router Direct Interconnection Router Router Backbone ISP Router Shared Traffic Exchange Point
9 Shortest Exit ( Hot Potato ) Routing Backbone ISP Shared Traffic Exchange Point San Francisco Los Angeles Chicago New York Washington DC Shared Traffic Exchange Point Web Site Dallas Backbone ISP Atlanta Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.
10 Backbone and Secondary Peering Backbone peering interconnects providers that have no need for a transit relationship. It provides the only interconnection between those providers. Secondary peering interconnects providers who would otherwise exchange traffic through some transit connection. Secondary peering is an economic optimization, reducing traffic over the transit connection.
11 Fundamental Economics of a Local or Regional ISP Larger ISP or Backbone Transit Connection Regional or Local ISP Concentration to a larger ISP or backbone provider with global connectivity by means of a concentrated, high bandwidth connection Many remote locations connect to a regional or local ISP with individual, low bandwidth connections
12 Secondary Peering Larger ISP or Backbone Larger ISP or Backbone Transit Connection Transit Connection Regional or Local ISP Regional or Local ISP The secondary peering connection will tend to exist if the cost of the connection to each ISP is less than the money each saves due to reduced transit traffic.
13 A Hierarchical View of the Internet Backbone Backbone Backbone Backbone Backbone Backbone Backbone Default Free Zone Backbone ISPs ISPs Customers Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.
14 A Hierarchical View of the Internet Upstream IBP Peering Connection IBP ISP ISP Downstream ISP cf. Lixin Gao
15 Negotiations for North American Backbone Internet Interconnection Typical US backbone interconnection guidelines Bi-coastal US presence, with multiple potential points of interconnection Significant transcontinental bandwidth Consistent routes at all locations Competent staff, professional 7 x 24 operation Rough balance of ingress/egress traffic Sufficient scale to justify transaction costs Where criteria are not met, a backbone may: decline to exchange traffic, OR expect cash or non-cash compensation in return Backbone providers negotiate traffic exchange terms and conditions on a case by case basis.
16 Emerging Global Trends The traditional hub and spokes system Traffic exchange trends in Europe Global deployment International diffusion of users and content
17 Traditional Hub and Spokes System Regional #2 Regional #1 Regional #3 NSP #1 ISP #1 ISP #3 Asian ISP ISP #2 NAP San Francisco Los Angeles NSP #2 NSP #3 Chicago NAP NAP New York Washington DC NAP ISP #4 Non-U.S. ISP pays vbns Dallas Atlanta Non-U.S. ISP pays European ISPs Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.
18 Former Hub and Spokes in Europe French ISP U.S. Backbone German ISP
19 Internet Traffic Exchange in Europe Today French ISP U.S. Backbone Interconnection Point German ISP
20 Factors Driving European Evolution Decline in street price of circuits within Europe due to deregulation. Declining cost of transoceanic capacity. Increased number and density of customers and content (and caching). Improved number and distribution of shared peering points. Deregulation of European telecoms, and recognition of the need to minimize regulatory barriers to Internet growth. New transit services terminated in Europe and elsewhere.
21 Global Internet Backbone Deployment Overseas Point of Presence (POP) European ISP pays Regional #2 Regional #1 Regional #3 NSP #1 ISP #1 ISP #3 Asian ISP ISP #2 NAP San Francisco Los Angeles NSP #2 NSP #3 Chicago NAP NAP New York Washington DC NAP ISP #4 Atlanta Backbone pays vbns Dallas Backbone pays Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.
22 International Diffusion 250 Number of Users Online Worldwide Millions of Users E United States Rest of World Source: IDC, Merrill-Lynch (Kende, FCC)
23 Motivations for International Pressure for Cost Sharing (ICAIS) Mistaken perception that U.S.-based backbones discriminate against overseas providers in our interconnection policies. Dissatisfaction with allocation of transoceanic circuit costs, which often are fully carried by the non-u.s.-based provider.
24 An Excerpt from Genuity s Guidelines 1. Presence at three or more Shared Interconnection Points listed above (two of which must be MAE-East ATM and MAE-West ATM), for Domestic ISPs; presence at two or more Shared Interconnection Points listed above for International ISPs. 2. For domestic ISPs, United States coast-to-coast nationwide backbone of at least 155Mbps. 3. Consistent route announcements at all exchange locations. 4. Experienced, professional Network Operations Center staffed 24x7. 5. Loose Source Record Route (LSRR) capability at core border routers on network. 6. For domestic ISPs, roughly balanced traffic. 7. A minimum Internet traffic exchange of 1 Mbps with Autonomous System Willingness to enter into a formal Internet Interconnection Agreement.
25 Economic Modeling of Internet Backbone Peering Analytical framework The off-net pricing principle Implications for international interconnection
26 Analytical Framework Define: c o as cost of origination c t as cost of termination a as an access charge levied on the sender Due to shortest exit, c t > c o Then cost for the originating network is c o + a cost for the terminating network is c t - a Network i Network j
27 Impact of Access Charges In a bill-and-keep system, access charges are zero. With equal access charges and symmetric traffic, net access charges are still zero. Providers will, however, view their marginal costs quite differently. (Laffont/Rey/Tirole)
28 The Off-Net Pricing Principle Under a broad range of conditions, marginal price should equal marginal cost. Providers will tend to charge the same for on-net traffic as for off-net traffic, i.e. there are no strong incentives for price discrimination.
29 Implications for International Interconnection Where a group of customers have distinguishable costs that are higher than those of other customers, the model predicts that marginal prices will be higher to exactly offset the higher costs. In the absence of market power, there is in fact no way for the ISP to absorb the cost difference without reflecting it in the price. This prediction is consistent with typical practices of USbased ISPs, both within the US and overseas.
30 Scaling Issues AS Number growth IPv4 address growth Routing table growth
31 Tony Bates s Data AS Growth Active ASes Food
32 Exponential Growth of Autonomous System (AS) numbers R2 = ASNs to Oct2000 ASNs since Oct2000 Expon. (ASNs to Oct2000) Source: Scott Marcus, Genuity 10/01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/2005 AS Numbers (ASNs)
33 Exponential vs Quadratic (Bates Data) y = 6E-14e x R 2 = ASNs to Oct2000 ASNs since Oct2000 Expon. (ASNs to Oct2000) Poly. (ASNs to Oct2000) y = 0.002x x + 2E+06 R 2 = /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/2003 Active ASes (Clean)
34 Chicken Little was Wrong! This is far simpler to remedy than IPv4 address exhaustion, because the solution need not impact end systems (hosts); the solution need not impact DNS; and the solution need not impact routers unless they speak BGP-4. Any solution is complicated by the need for backward compatibility and phased migration. Time until exhaustion is nonetheless sufficient to architect, design, implement and deploy solutions. Cisco and Juniper are reportedly well into implementation.
35 The RIRs Recognize the Need for Forecasting Continuing need to further refine projections. Need for forward-looking proactive forecasting on a regular basis not only for AS numbers, but also for route table entries and IPv4/IPv6 addresses. Forecasting needs to incorporate allocation data from all three RIRs (APNIC, ARIN, RIPE NCC). Forecasting needs to be institutionalized by the RIRs themselves, with data readily available to independent researchers.
36 The Team Assembled by ARIN Frank Solensky Gotham Networks kc claffy CAIDA Scott Marcus Genuity Active contributions and support by APNIC and RIPE NCC
39 Applications Have Distinct Needs is tolerant of delay, but every bit of data must ultimately be delivered correctly. For real time WWW traffic, delay must be low, but moderate variability of delay is permissible. Real time Voice over IP (VoIP) and video are tolerant of modest loss, but intolerant of large and variable delay.
40 Cross-Provider Differentiated Services Internet services today are best-efforts. Commoditized service, with marginal price equal to marginal cost. Differentiated services are of limited value today. Best-efforts are good efforts, so little incremental advantage for differentiated services. Customers have no demonstrated propensity to pay a premium for better CoS for data services. Better CoS for real time voice and video still interesting. CoS today is limited to a single provider s network.
41 Cross-Provider Measurements and SLAs Potentially valuable for best-efforts traffic; essential for Differentiated Services. Provider responsibility ends today at the peering interconnection point. No consistent measurement framework. No business arrangements to incent SLA adherence between peer networks. Information sharing of proprietary data is a difficult but not intractable problem.
42 Summary The distinction between peering and transit is vital to an understanding of the underlying business models that drive global Internet connectivity. The business models that have evolved around peering, in North America and globally, are complex but rational. Peering business relationships have adapted quickly in response to economic or regulatory stimuli, and will continue to evolve.
43 References Cremer, Rey and Tirole, Connectivity in the Commercial Internet, presented at Competition and Innovation in the Personal Computer Industry, San Diego, April, Marcus, Designing Wide Area Networks and Internetworks: A Practical Guide, Addison Wesley Longman, Chapter 14, which describes peering and transit, is available at Huston, Interconnection, Peering and Settlements, Laffont and Tirole, Competition in Telecommunications, MIT Press, General analysis of the economics of telecommunications pricing. Kende, The Digital Handshake: Connecting Internet Backbones, FCC, Gao, On Inferring Autonomous System Relationships in the Internet, IEEE GlobeCom 2000, San Francisco, November, Milgrom, Mitchell, and Srinagesh, Competitive Effects of Internet Peering Policies, presented at the American Economics Association, January, Laffont, Marcus, Rey and Tirole, Internet Interconnection, American Economics Review, May, 2001.
WP 11 01 ESMT White Paper ASSESSMENT OF A SUSTAINABLE INTERNET MODEL FOR THE NEAR FUTURE HANS W. FRIEDERISZICK, ESMT CA JAKUB KAŁUŻNY, ESMT CA SIMONE KOHNZ, ESMT CA LARS-HENDRIK RÖLLER, ESMT ISSN 1866-4016
BoR (12) 132 Differentiation practices and related competition issues in the scope of net neutrality Final report 26 November 2012 Executive summary... 4 1 Introduction... 11 2 The Internet value chain...
Interconnection and Peering among Internet Service Providers A Historical Perspective An Interisle White Paper. Prepared by: Interisle Consulting Group, LLC 39 S Russell Street Boston, MA 02114 http://
THE ECONOMICS OF THE INTERNET BACKBONE NICHOLAS ECONOMIDES New York University, New York Contents 1. Competition among Internet backbone service providers 1.1. Internet backbone services 1.2. Interconnection
Critical Business Decisions for Internet Services Joseph P. Bailey Ph.D. Candidate, Technology, Management and Policy Massachusetts Institute of Technology 77 Massachusetts Avenue, Bldg. E40-237 Cambridge,
TELECOMMUNICATIONS RESILIENCE GOOD PRACTICE GUIDE Version 4 MARCH 2006 Handling Instructions : This document was written for the use of the UK national infrastructure and it can be shared freely. Freedom
This work is distributed as a Discussion Paper by the STANFORD INSTITUTE FOR ECONOMIC POLICY RESEARCH SIEPR Discussion Paper No. 05-19 The Rise and Fall of Third-party High-speed Access By Gregory L. Rosston
DOCUMENTOS DE TRABAJO DOCUMENTO DE TRABAJO N 14 Connectivity in Latin America and the Caribbean: The role of Internet Exchange Points Hernán Galperin, Ph.D. Universidad de San Andrés/CONICET (Argentina)
US Business Services 2015 Executive Summary CMR Market Research May 2015 Reproduction without permission 1 The contents of this report represent CMR s analysis of the information available to the public
Essential Ingredients for Optimizing End User Experience Monitoring An ENTERPRISE MANAGEMENT ASSOCIATES (EMA ) White Paper Prepared for Neustar IT MANAGEMENT RESEARCH, Table of Contents Executive Summary...1
Committee on Payment and Settlement Systems The role of central bank money in payment systems August 2003 Copies of publications are available from: Bank for International Settlements Press & Communications
WOULD MANDATING BROADBAND NETWORK NEUTRALITY HELP OR HURT COMPETITION? A COMMENT ON THE END-TO-END DEBATE CHRISTOPHER S. YOO * ABSTRACT A chorus of commentators has drawn inspiration from the end-toend
Promoting Local Content Hosting to Develop the Internet Ecosystem Michael Kende and Karen Rose January 2015 With thanks to Executive Summary In many developing countries and emerging regions the vast majority
Flat Rate versus Per Minute Charges for Telephone Service: The Relationship between Internet Access and Telephone Tariffs December 4, 2001 The vast majority of residential Internet users and many business
MANAGEMENT S DISCUSSION AND ANALYSIS Dated: July 9, This Management's Discussion and Analysis ("MD&A") for the three months ended (second quarter of fiscal ) provides detailed information on the operating
Call Termination Fees: The U.S. in global perspective 1 Abstract The economic framework under which the United States implements call termination fees is unusual. Several recent studies suggest that the
VoIP Impairment, Failure, and Restrictions A BROADBAND INTERNET TECHNICAL ADVISORY GROUP TECHNICAL WORKING GROUP REPORT A Uniform Agreement Report Issued: May 2014 Copyright / Legal Notice Copyright Broadband
Specialized Services: Summary of Findings and Conclusions Specialized Services Working Group Open Internet Advisory Committee Federal Communications Commission Released August 20, 2013 Full Annual Report
IP TELEPHONY POCKET GUIDE BY BARRY CASTLE 2nd Edition September 2004 ShoreTel, Inc. 960 Stewart Drive Sunnyvale, CA 94085 408.331.3300 1.800.425.9385 www.shoretel.com email@example.com TABLE OF CONTENTS
How to Successfully Select and Implement a Hosted VoIP System Written by Igal Rabinovich, CEO IT Help Central How to Successfully Select and Implement a Hosted VoIP System Page 1 TABLE OF CONTENTS EXECUTIVE
Industrial Ethernet: A Control Engineer s Guide Abstract As part of a continuing effort to make their organizations more efficient and flexible, manufacturers are rapidly migrating to Industrial Ethernet
Risk perception and risk management in cloud computing: Results from a case study of Swiss companies Nathalie Brender Haute Ecole de Gestion de Genève Campus de Battelle, Bâtiment F 7 route de Drize, 1227
WHITE PAPER Managed Services for Mission Critical Communications Introduction Much has been written about Managed Services in the communications industry, initially in the IT sector and more recently in
BGP routing policies in ISP networks Matthew Caesar UC Berkeley Jennifer Rexford Princeton University Abstract The Internet has quickly evolved into a vast global network owned and operated by thousands
December 2008 VoIP YOUR GUIDE TO VOICE OVER IP 2 An Advertising Supplement to USA Today CONTENTS 2 Foreword from TIA 4 What can VoIP do for me? 4 Great savings and features 5 Savings without the maintenance