Overview of recent changes in the IP interconnection ecosystem

Transcription

1 Public report Overview of recent changes in the IP interconnection ecosystem May

2 Contents 1 Executive summary Internet trends Evolution of Internet interconnection 2 2 Introduction 4 3 Internet trends Internet traffic has grown exponentially Internet traffic has become globalized The value of Internet content and applications has increased dramatically Bandwidth-intensive types of Internet traffic have become predominant Cloud computing services have become widely utilized Internet connections are a key feature for many new devices and services Conclusion 18 4 Evolution of Internet interconnection Description of the early Internet Internet exchange points (IXPs) Internet service providers (ISPs) Content providers Impact on backbone providers 28 5 Conclusion 34 Annex A: Introduction to the IP interconnection ecosystem Annex B: About Analysys Mason

3 Copyright The information contained herein is the property of Analysys Mason Limited and is provided on condition that it will not be reproduced, copied, lent or disclosed, directly or indirectly, nor used for any purpose other than that for which it was specifically furnished. Analysys Mason Limited 818 Connecticut Avenue NW Suite 300 Washington DC USA Tel: (202) Fax (202) Registered in England: Analysys Mason Limited Bush House, North West Wing, Aldwych London WC2B 4PJ, UK Reg. No

4 Overview of recent changes in the IP interconnection ecosystem 1 1 Executive summary Last year saw the 15th anniversary of the commercialization of the Internet backbone as we know it today, when the US National Science Foundation Network (NSFNET) Backbone Service was decommissioned in favor of the current commercial Internet backbone. On that day, April 30, 1995, the Netscape browser had just been introduced, AOL was the most popular ISP in the US and was still charging hourly fees, and the DOCSIS standard for cable modem Internet access had not yet been released. The Internet was mainly used for file transfer, was simple text, and the Web was not yet multimedia. Over the intervening 15 years, there have been significant changes in the Internet ecosystem, including an enormous increase in Internet traffic and bandwidth along with a corresponding dramatic increase in the value generated by Internet content. During this period, interconnection arrangements between all of the players in the ecosystem have continuously evolved to meet the challenges of the changing Internet ecosystem, driven by commercial considerations and not regulation Internet trends The changes in the Internet ecosystem are mainly due to the following interrelated trends: Internet traffic has grown exponentially Internet traffic has become globalized The value of Internet content has dramatically increased There has been an explosion of Internet access and in the traffic generated by each individual user, for the reasons described below. As a consequence, market players have adapted by modifying their network architectures and revamping their relationships with suppliers and peers. From its historical origins in the US, Internet usage has grown significantly around the world, and correspondingly, Internet content creation has also globalized. As a result, the proportion of Internet traffic originated and terminated outside the US has significantly increased over time. Additionally, the amount of Internet traffic exchanged in US-based Internet exchange points (IXPs) is now much less predominant, as many countries and regions have expanded their Internet traffic exchange capabilities through one or more local IXPs. 15 years ago, money flows were mostly directed to ISPs and backbone providers for Internet access and traffic, while content was mostly free or had limited value. This has changed significantly, as content has gained in value thanks to an improved quality of access and the corresponding availability of premium content. As a result, content providers have been able to leverage the value of their content in order to reduce their costs for 1 The only instances of regulatory intervention in IP interconnection have been during merger proceedings, when agencies have imposed conditions ranging from divestiture to temporary limits on changes in peering agreements.

5 Overview of recent changes in the IP interconnection ecosystem 2 delivering their traffic (for instance, by building their own infrastructure to distribute content around the country closer to end-users). Bandwidthintensive and quality needy types of Internet traffic such as video have become predominant Cloud computing services have become widely utilized Internet connection became a mandatory feature for many new consumer electronic devices Bandwidth-intensive and quality needy types of traffic, such as video, have experienced a massive growth, which has had a major impact on bandwidth requirements and the direction of traffic flows. There has been a significant increase in the amount of traffic being downloaded from centralized sites such as YouTube, while at the same time the popularity of peer-to-peer services has resulted in significant traffic between end users. These increases in overall traffic, along with changes in traffic patterns, have led ISPs and content providers to adapt their network policies and architectures. Cloud computing services, ranging from online to business applications, have been embraced as barriers to their adoption have been progressively lifted, thanks to the improvement of Internet access, the development of security safeguards, and the resulting increase in awareness and confidence in cloud computing. This has increased overall usage of the Internet, while also making consumers more sensitive to latency and other quality measures that impact access to their cloud applications. Many new devices have emerged that incorporate Internet access: mobile devices with features adapted to data traffic, netbooks and tablets designed to enable Internet access to cloud-computing applications, and new video streaming devices relying on broadband. As a result, many traditional communications services have migrated to the Internet, for example TV over IP and Voice over IP (VoIP). This has led market players to increase mobile access, install additional network capacity, and establish new forms of interconnection. The changes described above, individually and in sum, have led to significant changes in the underlying Internet ecosystem as it was established 15 years ago. 1.2 Evolution of Internet interconnection 15 years ago, the Internet ecosystem had a relatively simple structure, with clear and hierarchical relationships between end users and content providers buying Internet access from ISPs, ISPs buying Internet transit from backbone providers, and backbone providers peering with each other. 2 At the time, given the historical development of the Internet, much of the global traffic originated, terminated, or transited the US. But Internet players have steadily adapted to the changing market conditions described above in a number of ways, in order to carry greater traffic volume with lower latency, and to reduce their transit 2 For an introduction to Internet market players and interconnection definitions, please see Annex A.

6 Overview of recent changes in the IP interconnection ecosystem 3 costs, thereby changing the traditional hierarchy of the ecosystem in the process, while also making it truly global. Internet Exchange Points (IXPs) facilitated increased connectivity ISPs and content providers increasingly connect directly Content providers have a variety of options for delivering traffic The development of IXPs around the world has benefited backbone providers by facilitating distributed interconnection between peers and transit customers, which improves the quality of service and reduces traffic carrying costs. However, the increase in utilization of IXPs also helped in marginalizing the role of the backbone providers, as former customers began to interconnect directly with one another at the IXPs. Many route around options have emerged for ISPs and content providers to exchange Internet traffic, while avoiding Internet backbone transit costs. These include secondary peering arrangements between ISPs to exchange traffic directly, much of it from peer-to-peer applications. Increased traffic has also led content providers to negotiate paid peering arrangements with ISPs to deliver content to their end users. Backbones have adapted to these changes by selling only partial transit to ISPs and content providers, which have arranged much of their traffic exchange directly. At the same time, peering relationships have evolved based on volume, traffic flows and routing. Increases in content traffic have led to traffic imbalances, such that ISPs delivered much more traffic from content providers to their end users. This has led to paid peering arrangements to cover the cost of the traffic. Content providers have leveraged their expanding financial resources to deliver traffic directly to ISPs by building private networks or using third-party content delivery networks (CDNs) to deliver content directly to caches at the edge of ISPs networks, further reducing transit costs and the resulting revenues for backbones. From the backbone provider s point of view, these changes led to a reduction in demand for transit services, and an increase in competition from former customers who now have a number of choices for delivering and exchanging traffic. Further, backbones must compete vigorously on the price of transit in order to generate the traffic volume to continue to peer with one another. This has resulted in an increase in the level of competition for Internet transit services, as evidenced for example in the fall in transit prices over the past five years, with no sign of respite. In conclusion, in the 15 years since the commercialization of the Internet backbone, the Internet ecosystem has proven itself to be able to develop and sustain interconnection in the absence of sectorspecific regulation. It has also shown itself to be able to adapt well to rapid and profound market changes without regulatory intervention, which in telecommunications is typically much slower to implement changes in interconnection arrangements and has issues with implicit subsidies and arbitrage. As a consequence of these changes, today s Internet ecosystem is no longer hierarchical, but rather a dynamic web of interconnections between a variety of Internet players.

7 Overview of recent changes in the IP interconnection ecosystem 4 2 Introduction This paper marks the 15th anniversary of the commercialization of the Internet backbone as we know it today. There have been significant changes in the Internet ecosystem over the last 15 years, resulting in enormous increases in Internet traffic, supported by a massive expansion of the Internet bandwidth, and accompanied by a major increase in the value generated by Internet services and applications. Through these changes, one feature remains prominent : interconnection arrangements between all of the players in the ecosystem have continued to evolve to meet the challenges of the changing ecosystem, driven almost entirely by commercial considerations and not regulation. 3 On April 30, 1995, the US National Science Foundation Network (NSFNET) Backbone Service was decommissioned in favor of the current commercial Internet backbone. The NSFNET operated a national backbone network allowing regional networks to connect at supercomputing sites. 4 At this time, the objective was to create an open network linking academic researchers, and allowing them access to distant supercomputers over the network at no cost. When the NSFNET network went online in 1986 and allowed commercial traffic (and not just NSFNET traffic) to be carried, it consisted of six sites interconnected with leased 56kbit/s Digital Data System (DDS) links this was upgraded to 45Mbit/s links in At the time that the NSFNET was decommissioned in 1995, Netscape the first commercial Web browser had just been introduced, AOL was the most popular ISP in the US and was still charging hourly fees, and the DOCSIS standard for cable modem Internet access had not yet been released. The Internet was used mainly for file transfer, was simple text, and the Web was not yet multimedia. Since then, traffic has exploded, as shown in Figure 2.1below. 3 The only instances of regulatory intervention in IP interconnection have occurred during merger proceedings, when agencies have imposed conditions ranging from divestiture to temporary limits on changes in peering agreements. 4 From 1987 to 1995, the NSFNET Backbone was designed, managed, and operated on behalf of the NSF by Merit Network, Inc., a nonprofit corporation governed by public universities partnering with IBM, MCI, and the State of Michigan.

8 PB per month TB/ month Overview of recent changes in the IP interconnection ecosystem 5 2,500,000 2,000,000 1,500,000 Figure 2.1: Monthly backbone traffic in the US [Source: Minnesota Internet Traffic Studies; Analysys Mason Estimates] 1,000, ,000 0 This increase in traffic comes as a result of a virtuous circle of more powerful computers, the availability and adoption of broadband access, the increase in bandwidth available for carrying Internet traffic, and the introduction of multimedia content (particularly video), along with new ways to share that content, such as peer-to-peer. Figure 2.2 below shows how new categories of Internet video have arisen in the last five years and are projected to become the dominant category of content over the next few years. 12,000 10,000 8,000 Figure 2.2: Consumer Internet traffic forecasts [Source: Cisco Visual Networking Index] 6,000 4,000 2, Web and Internet Gaming Internet Video com. Internet Video to TV File Sharing Internet Voice Internet Video to PC Note: in the figure above, the File sharing category includes traffic from peer-to-peer applications such as BitTorrent and edonkey, as well as web-based file sharing. The NSFNET was the backbone over which regional networks exchanged traffic. When it was closed in favor of commercial backbones, there was no longer a single backbone that could be used for

9 Overview of recent changes in the IP interconnection ecosystem 6 traffic exchange. Instead, four Network Access Points (NAPs) across the country were designated as the points at which traffic could be exchanged by the new commercial backbones. Such interconnection was not regulated, making it possible for commercial arrangements known as peering and transit to be negotiated between the backbones and ISPs. The introduction of commercial backbones not only modified the structure of the Internet infrastructure ecosystem, but also accommodated to changes to the type of Internet traffic carried, as this opening increased greatly the carrying and exchange of commercial traffic in addition to the university and research traffic previously carried. This document aims at describing changes in the Internet ecosystem since the commercialization of the Internet backbone. We analyze the changes in terms of the type of Internet traffic carried, the infrastructure supporting the Internet traffic, as well as the evolution of the different players and their inter-relationships. We also assess how commercial models adapted to these changes, and analyze the impact of these changes on competition between the backbones. The remainder of this document is laid out as follows: Section 3 provides an outlook of the main trends over the past 15 years that have impacted the Internet backbone Section 4 analyses the evolution of the Internet backbone, with a focus on interconnection arrangements, and describes the current market situation from point of view of each of the main types of Internet players Section 5 summarizes the main conclusions reached.

10 Overview of recent changes in the IP interconnection ecosystem 7 3 Internet trends The Internet ecosystem has evolved significantly over the past 15 years, with corresponding impacts on interconnection arrangements. This evolution is partly due to a number of trends that are all interrelated, but that can be described as follows: Internet traffic has grown exponentially Internet traffic has globalized the value of Internet content and Applications has increased dramatically bandwidth-intensive types of Internet traffic such as video have become predominant cloud-computing services have become widely utilized Internet connections are a mandatory feature for many new consumer electronic devices and services. Peer-to-peer and video streaming Online devices Figure 3.1: Trends that have impacted the Internet ecosystem over Volume of Internet traffic the past 15 years [Source: Analysys Mason] Globalization of Internet traffic Value of Internet Content Cloud computing We provide in this section an analysis of these trends, which will help to better understand the context of the evolution of Internet interconnection described in the following section. 3.1 Internet traffic has grown exponentially The past 15 years have seen an explosion of Internet usage around the world, particularly from Internet broadband subscriptions that allow end users to use their connections to access the new high bandwidth services such as video streaming. At the end of September 2009, there were more than 75 million fixed broadband subscriptions in the US, representing a penetration of around 60% of households, and significant growth from the end of 2001, when there were only around 10 million broadband subscribers, as shown in the Figure 3.2 below.

11 Millions. Penetration as % households Overview of recent changes in the IP interconnection ecosystem % 90% 80% 70% 60% 50% 40% 30% 20% 10% Figure 3.2: Growth in number of broadband subscribers in the US [Source: Analysys Mason Research] Q3 0% DSL Residential FTTB Broadband penetration Cable modem Broadband FWA In the past 10 years, this dramatic increase in the adoption of fixed broadband has been accompanied by an explosion of mobile broadband usage. According to CTIA, wireless data revenues in the US rose from US$211 million in 2000 to US$41.5 billion in Similar growth has characterized markets in Europe and much of Asia as well, resulting in demand for Internet services that is distributed much more evenly across the world than 15 years ago, as shown in Section 3.2. Not only has the number of Internet access users experienced a significant growth, but also the bandwidth usage by each individual user has exploded. This can be explained in part by the increase in the availability of even higher capacity Internet links, and the growing availability of bandwidthintensive applications such as video downloads (see section 3.4), the development of cloud-computing services (see section 3.5), and the take-up of a large variety of devices connected to the Internet (see section 3.6). These combined effects have resulted in an exponential growth of Internet traffic in general, and in the traffic carried by Internet backbone providers in particular, as shown above in Figure 2.1. The fast growth in traffic has naturally generated significant changes in the Internet ecosystem, as market players have adapted to this demand increase by modifying their network architectures (e.g. expanding their networks by deploying fiber and upgrading their capacity to accommodate higher traffic volumes and traffic that is more latency sensitive, and creating new routes to carry the traffic more efficiently), as well as revamping their relationships with suppliers and peers, as described below. 5 Source:

12 Overview of recent changes in the IP interconnection ecosystem Internet traffic has become globalized For historical reasons, in the early 90s Internet usage was biased towards the US, as was Internet traffic exchange at the time that the NSFNET was decommissioned in This situation evolved significantly over time, as demand for Internet access exploded first in all the other developed countries, and then in developing countries. Indeed, today a significant portion of Internet users are to be found in large markets like China and India: for example, China today boasts more than 400 million Internet users, and other emerging Asian nations are also fuelling the growth of traffic in this region. 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Figure 3.3: Evolution of the distribution of Internet users by region[source: ITU] North America East Asia and Pacific Middle East and North Africa Sub-Saharan Africa Europe and Central Asia Latin America and Carabbean South Asia The increase in the number of Internet users outside the US led to a corresponding increase in Internet traffic in other regions of world, with Internet content being created, hosted and accessed from outside the US. As a result, the Internet network topology began to evolve, so as to enable a more efficient carrying of traffic. The historical use of US-based IXPs to exchange foreign traffic was not only inefficient, but also led to a call for intervention that went by the rubric of International Charging Arrangements for Internet Services (ICAIS). In this debate at the Asia-Pacific Economic Cooperation (APEC) forum and the International Telecommunication Union (ITU), countries underserved by Internet infrastructure sought to address the issue of bearing the costs of connecting to US-based IXPs for traffic exchange. To deal with these issues, national and regional networks started to interconnect to each other and exchange their traffic at regional IXPs such as AMS-IX in Amsterdam and LINX in London. These IXPs exchange a significant amount of traffic among a growing number of members. For instance, AMS-IX has 389 connected networks exchanging Gbit/s of traffic at peak times, while LINX

13 kbit/s Overview of recent changes in the IP interconnection ecosystem 10 has 379 members with a peak traffic exchange rate of 614Gbit/s. 6 This traffic is not just national, but also international, and the result is that the host countries have turned into significant international hubs. As shown in Figure 3.4 below, the Netherlands, followed by the UK, has a distinct edge in the amount of international Internet bandwidth per population compared with a wide variety of countries, based in no small part on the significant role of Dutch and British IXPs in attracting international backbones and content providers. Similar activity is taking place in Asia and to a lesser degree in other parts of the world, as IXPs act to help to localize traffic and attract a surrounding ecosystem of backbones and content providers Australia USA Canada Germany France Singapore UK Netherlands 00 Figure 3.4: Growth in international Internet bandwidth per head of population [Source: Telegeography, ITU, Euromonitor, company data] Consequently, the role of the US as a hub for international exchange of Internet traffic has diminished significantly. Figure 3.5 below shows that even though Latin America still relies heavily on the US for Internet bandwidth, regional networks in Europe, Asia and Africa are less and less dependent on US-based facilities to carry and exchange traffic. Africa s Internet traffic is now focused more predominantly on Europe, while Europe s traffic is mainly intra-regional, which is also becoming the norm in Asia. 6 In October 2010 AMS-IX had 389 networks connected, measured by autonomous system numbers, with a peak traffic exchange of Gbpit/s (source: LINX provides general statistics showing that in October 2010 it had 379 members exchanging 614Gbit/s peak traffic (source:

14 Overview of recent changes in the IP interconnection ecosystem % 90% 80% 70% 60% Figure 3.5: Share of Internet bandwidth connected to US and Canada, by region [Source: Telegeography] 50% 40% 30% 20% 10% 0% Africa Asia Europe Latin America Figures represent Internet bandwidth connected across international borders as of mid-year. Domestic routes are excluded. Today, 89 countries possess their own IXPs to exchange national and regional Internet traffic, and thus lower the costs associated with international Internet access. 7 It is interesting to note that this evolution has not required specific regulatory intervention, but is the result of commercial solutions adopted by national and regional networks. 3.3 The value of Internet content and applications has increased dramatically As new Internet services have developed, the positioning of the different stakeholders in the Internet ecosystem has evolved (see Annex A for an introduction to the IP interconnection ecosystem). 15 years ago, ISPs and backbone providers were at the top of the hierarchy, and money flows were mostly directed to them for Internet access and traffic, while content was mostly free or of limited value. This has changed significantly in recent years as the value of content has risen, thanks to an improved quality of service and the development of premium content. Online music provides a good illustration of the evolution of content values in the past decade. Initially, music labels were reluctant to sell their music online, for a variety of reasons. As a result, the first widespread distributor of online music was the peer-to-peer network Napster. This was free, but content was largely pirated and copyright issues led to its shutdown in Following this, labels became comfortable with the level of protection being offered by digital rights management (DRM), and began to make their music available online, leading to the emergence of paid downloads. A good 7 Source: Packet Clearing House Report on Internet Exchange Point Locations (

15 USD million Overview of recent changes in the IP interconnection ecosystem 12 illustration is the success of the media platform itunes Store, launched by Apple in April 2003: as of February 2010, 10 billion songs have been sold on this platform. 8 Other significant value-added services have emerged from mobile Internet access, with the explosion of the mobile paid applications market. The success of Apple s App Store can illustrate this trend: as of January 2010, 3 billion applications (both free and paid-for) had been downloaded through the store. 9 More generally, the Yankee Group estimates that the total revenues for the mobile paid applications industry in the US alone reached US$1.6 billion in 2010, against US$537 million the preceding year. This rise in content value will continue in the coming years, as this market growth will be supported by devices such as the new ipad designed for the consumption of online multimedia content. According to the same source, application revenues are forecast to reach US$11 billion by The massive growth in Internet advertising revenue illustrates well the growth in content value. Figure 3.6 below provides an estimate of the evolution of the fixed and mobile Internet advertising markets in North America. The Internet advertising market is estimated to have been worth around US$25 billion in 2009, compared with approximately US$10 billion five years before, and is projected to grow to more than US$35 billion by It should be noted that a large part of these revenues are captured by content providers such as Google. 40,000 35,000 30,000 25,000 Forecasts Figure 3.6: Internet advertising market in North America [Source: PricewaterhouseCoopers LLP, Wilkofsky Gruen Associates] 20,000 15,000 10,000 5,000 0 Wired Internet advertising Mobile Internet advertising 8 Source: 9 Source : 10 Source :

16 Overview of recent changes in the IP interconnection ecosystem 13 An interesting example of the evolution of the value of content is ESPN360.com, a broadband network for live sports programming in the US and several other countries. ESPN has essentially adopted a pay-tv model, in which ISPs must sign up and pay ESPN for offering the service, rather than charging end users directly. The customer pays for access as a part of its ISP fees, whether it uses the service or not, and may choose an ISP based on the availability of ESPN360. This example suggests the extent to which the relative positions of content providers and ISPs have evolved in the Internet ecosystem. In particular, content providers have been able to leverage the attractiveness of their content in order to reduce their supply costs for accessing the Internet. Furthermore, as we will see in Section 4.4, content providers can use their fast-growing revenues to invest in their own network infrastructures, in order to further reduce the transit costs by developing self-supply solutions, and thus avoid paying other market players to deliver their traffic. 3.4 Bandwidth-intensive types of Internet traffic have become predominant The nature of Internet traffic has changed dramatically in recent years, and in particular there has been a massive growth in the proportion of video traffic as a percentage of total Internet traffic. Video traffic is distributed either via peer-to-peer services or streamed from a centralized server. Projections show that video will remain a major driver for traffic increase (see Figure 2.2) and sustain the growth in Internet traffic. We focus in this section on the growth of streaming video and peer-to-peer traffic, which not only has a major impact on the bandwidth requirement for Internet traffic, but also on the traffic patterns, as video streaming modifies the traffic pattern from traditional two-way communication to one-way content delivery) Sustained growth in video traffic The consumption of video media consists mostly of Internet video to PC (i.e. free or pay-tv viewed on a PC), and in a more limited scale in Internet video to TV (the same as video to PC, but streamed to the TV set). Thus today, 82% of Internet users in the US watch videos online, and an average user views 182 videos per month. 11 YouTube, a video-sharing website on which users can upload and view videos, is a leading symbol of this Internet explosion. YouTube was founded in February 2005 and bought by Google in In only five years, this company achieved an almost unimaginable growth: by October 2009, it was serving more than a billion videos per day worldwide. 12 In November 2009, 12.2 billion videos were viewed on YouTube on a monthly basis in the US alone. 13 Not only has YouTube managed to draw a very large audience, it has also succeeded in attracting a large amount of free user-generated content: YouTube announced on March 2010 that users were uploading some 24 hours of video to the platform every minute. 14 This represents double what was 11 Source : 12 Source : 13 Source: 14 Source:

17 Hours Q Q Q Q Q Q Q Q Q Q Q Q Overview of recent changes in the IP interconnection ecosystem 14 uploaded only two years before (of course, some of this material is commercial, and YouTube has had to address the copyright issues that arose first with Napster) Figure 3.7: Hours of video uploaded on YouTube platform per minute [Source: YouTube] But YouTube, while significant, is not the only player in the Internet video market: there are other major video content providers that are experiencing similar growth levels. Another example would be Hulu, a website offering commercial-supported streaming video of TV shows and movies from several major TV channels and studios. In November 2009, Hulu was streaming 924 million videos per month in the US alone, according to the research company Comscore. 15 In the UK, the BBC s online iplayer is estimated to have received well over a billion requests for programming in 2010, with 114 million requests in July 2010 alone. 16 Other services such as Netflix stream video directly to the TV. According to Sandvine, streamed audio and video represent almost 43% of data consumption in North America during peak periods, with Netflix video alone accounting for over 20% of download traffic during peak times. 17 In addition to these video-streaming sites, a significant amount of video is consumed using peer-to-peer networks. 15 Source: _Billion_for_First_Time_on_Record 16 This includes both video and radio programming. See 17 Source: Fall 2010 Global Internet Phenomena Report, Sandvine, p.9.

18 Overview of recent changes in the IP interconnection ecosystem Sustained growth in peer-to-peer traffic Peer-to-peer networks allow a distributed community of users to share digital content or resources, including video. 18 But contrary to a classic client server architecture in which all network content is held on a server in a central location and accessed via a client on the end-users devices, peer-to-peer resources are located in and provided by devices at the edge of the network by peers. Peer-to-peer quickly became a popular way for Internet users to share video and music files, as it was particularly adapted to transfer large files over the Internet. Indeed peer-to-peer networks use the resources provided by users (bandwidth, storage space, and computing power) to increase the total capacity of the system at little or no cost to the originator of the peer-to-peer network. In contrast, in a typical client server architecture, clients make demands on the system, but do not supply any resources (as such, as more clients join the system, the owner must provide more resources to meet the demand, or congestion will impact performance). Peer-to-peer services themselves are not a recent Internet application: they have their origins in many of the early Internet services, such as Internet Relay Chat (IRC) which was developed at the end of the 1980s. The widespread popularization of peer-to-peer goes back to Napster in Napster s success was striking, as only two years after the launch of operations it was reaching 26.4 million users. As much of the material distributed over Napster violated copyrights, the service was closed down after a lawsuit instigated by the Recording Industry Association of America (RIAA). 19 Nonetheless, peer-topeer networks have become more and more popular, and have overcome the perceived drawbacks of Napster s network (centralized servers distributing the files and requests to the peers) by relying on a decentralized or a hybrid architecture: without central servers, the technical risks are minimized, operational costs are reduced (database management in particular is distributed to peers) and the ability of authorities to shut down the service following legal decisions is much more limited. Kazaa and Gnutella 0.6 are notable examples of such new peer-to-peer networks. 20 As broadband access grew, and music began to shift to commercial sites such as itunes, peer-to-peer networks were increasingly used for the exchange of higher-bandwidth video traffic. The result of the popularization of peer-to-peer was a significant change, in which users became content providers as well as consumers, and the flow of content shifted from a centralized location to become more distributed. Unlike the streaming of a video from a centralized website such as YouTube, the downloading of the same video using a peer-to-peer network has a very different impact in terms of network utilization: in this case the traffic goes from one or more end users to another end user (and both ways, as peer-to-peer is based on reciprocal sharing) Even though the rise in peer-to-peer traffic is relatively recent, peer-to-peer services themselves should not be considered to be a recent Internet application. Peer-to-peer networks have their origins in many of the early Internet services: for instance, the Internet Relay Chat (IRC), which was developed at the end of the 1980s. 19 See 20 See 21 The difference may be most clearly understood by comparing YouTube with a peer-to-peer network. In either case, the content may be user-generated, but with YouTube the originator uploads it once to YouTube, from which others download it. In contrast, with peer-topeer, each time a user downloads the content it is resent from the originator (or other peers).

19 Overview of recent changes in the IP interconnection ecosystem 16 As a big proportion of traffic goes directly from ISP end users to ISP end users, and large file-sharing traffic requires very high bandwidth, ISPs have responded by adapting their network policies and architectures. In particular, they have had to find alternative routes to deliver this video traffic. Combined with the general growth of Internet traffic, it started to make economic sense for ISP to adopt solutions like secondary peering to address this traffic issue (see Section 4) Forecasts Growth in video traffic is not expected to slow down in the medium term, and according to Cisco, the sum of all forms of video (IPTV, video on demand, Internet video, and peer-to-peer) will account for over 85% of global consumer traffic by Internet video excluding peer-to-peer (free or pay TV and video on demand delivered on a PC or a TV) will account for over 55% of all consumer Internet traffic by this time. 22 It appears therefore that video traffic will be the major driver for the Internet traffic growth in the coming years, in proportions that will require significant investments in Internet infrastructures and technology development to carry the traffic and sustain the quality of service (via an increase in available bandwidth). It is interesting to note that in relative terms peer-to-peer is actually declining as a percentage of overall IP traffic, in favor of video streaming from centralized servers. This evolution can be explained by the growing availability of alternative ways to watch online premium content (e.g. YouTube, iplayer and Hulu.com) using centralized servers, and the relative drawbacks of peerto-peer downloads compared to video streaming. From video-streaming websites, content is available with high quality, is often free, and can typically be downloaded quickly and easily. By contrast, peerto-peer content, while also free, is not immediately available (content is always downloaded in random chunks, which makes it difficult to estimate the download completion time), there is no quality guarantee, and users are subject to an additional risk resulting from possible copyright violations. 3.5 Cloud computing services have become widely utilized Online services have also emerged significantly over the past 15 years: examples include data storage, and remote applications ranging from or online photo albums, to suites of business applications including spreadsheets and word processing. Because the content and applications are centralized in the network, rather than distributed at its edges, this is often referred to as cloud computing. 23 Although the concept of cloud computing is not new, it has become more popular recently as the barriers to take-up have been progressively lifted, thanks to the improvement of Internet access (e.g. development of broadband, availability of cheap dedicated high capacity links), the improvement of 22 Source: 23 As defined by the National Institute of Standards and Technology, cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. (See

20 Overview of recent changes in the IP interconnection ecosystem 17 security (in particular, enhanced encryption techniques), and the increase in awareness and confidence in cloud-computing techniques. YouTube is a good example of a popular cloud-computing application, where users store videos on YouTube s servers, and access them remotely in a streaming mode. The services offered by Microsoft and others can also be seen as a very successful popularization of cloud-computing services. For instance, Google s Gmail became available to the general public on February 7, 2007 (after an initial invitation-only beta release on April 1, 2004), and by December 2009 had already gained 176 million users. 24 This began a transformation of services that were formerly performed at the edge of the network and perhaps involved some incidental Internet traffic, into ones that were performed in the network itself, with integral Internet access and only intermittent processing at the edge. In addition to the examples discussed above, applications such as word processing and spreadsheets began to be offered as Internet services, threatening traditional software models. From the perspective of this paper, this increased overall usage of the Internet, and also made consumers more sensitive to latency and other quality measures that impacted their access to their cloud applications. 3.6 Internet connections are a key feature for many new devices and services The past 15 years have seen the development (and the wide acceptance by the general public) of a large variety of devices that are connected to the Internet to enable access to the new services described above, as well as additional services that were unleashed in response to the new usage patterns that resulted from these devices. New connected devices The release in 2007 of the first popular smartphone Apple s iphone marked the take-off of mobile Internet access. More and more mobile devices then started to offer specific features adapted to Internet browsing and data traffic (e.g. mobile applications). By June 2009, there were approximately 6.4 million active iphone users in the US, with 88% of iphone users accessing the Internet and 75% of them downloading mobile applications. 25 According to Validas, iphone users in the US each consumed 273MB of data per month on average, and 12% of iphone users consumed at least 500MB of data per month (as of February 2010). 26 Most vendors today are offering similar large touch-screen devices with embedded Internet functionalities. Netbooks are getting a mass-market status, and according to the research firm ABI Research, 35 million netbooks were shipped worldwide in Netbooks are specifically designed to connect 24 Source: 25 Source : 26 Source : 27 Source :

21 Overview of recent changes in the IP interconnection ecosystem 18 remotely to the Internet, either via a WiFi connection or a 3G connection, and are thus optimized for cloud computing with relatively little memory for installed software. Tablets can also support new usage intimately related to Internet connection, and the launch of Apple s ipad and other similar products supports the view that expectations are high for this market, with more than 14 million ipads being sold in New video-streaming devices have also emerged that exclusively rely on a broadband connection: examples include Vudu, the Boxee Box (so-called Social Media Center ), etc. These boxes directly enable video streaming to the TV, and video file-sharing in competition with pay-tv, and are beginning to be embedded into other devices such as game consoles or TV sets. New online multimedia services Second, many new services related to communications and multimedia have been released that substitute traditional telecom services with online equivalents. This evolution can be explained by the fact that an embedded Internet connection can enable advanced features and reduce the costs of carrying traffic. In addition to these services offering video on demand, broadcast TV services can also be provided over an Internet connection (IPTV), in particular as part of the triple-play offerings of telecom operators. Voice over IP (VoIP), which consists in delivering voice communication over IP networks, has a tremendous potential, as it allows customers to make free or very cheap voice calls. So far, quality issues have restrained a wide acceptance of VoIP, but increases in fixed network capacity are changing the situation at a rapid pace. Moreover, the development of cheap mobile broadband offers ( unlimited data plans with mobile subscriptions) opens the way to VoIP over mobile: for instance, the Line2 application for iphone enables users to call using the data connection instead of the traditional minutes included in the bundle and paid for by the user. In summary, the development of electronic devices and services relying on Internet access has generated significant changes in the Internet ecosystem, as market players have adapted to this demand increase by modifying and investing in their network architectures: rolling out mobile broadband networks, installing additional network capacity, and developing new forms of interconnection as described below. 3.7 Conclusion As explained previously, and illustrated in Figure 3.8, Internet traffic and revenues have been fed by the simultaneous growing demand for Internet-based services and the development of Internet-based solutions to answer and sustain this demand. 28 Source: Apple financial results

22 $ per Internet user per month. Overview of recent changes in the IP interconnection ecosystem 19 Willingness to pay for content Demand Internet traffic and revenues Mobile broadband Peer-topeer Figure 3.8: Supply and demand feed the growth of Internet traffic and revenues [Source: Analysys Mason] Netbooks Supply Cloud computing Video streaming It is not only Internet traffic that has exploded over the past 15 years: so has the Internet market as a whole. It is interesting to note that, while there has been a significant increase in the revenues generated by the e-commerce market and to a smaller scale by the online advertising and the device manufacturing markets there has been a more measured increase in revenues from Internet access itself. As shown in Figure 3.9 below, the revenues per month per Internet user generated by e- commerce and online advertising significantly outweigh those from Internet access Online advertising E-Commerce Devices Internet access Figure 3.9: Evolution of the Internet ecosystem in terms of revenues per Internet user per month [Source: PWC, US Department of Commerce, Bureau of economic analysis, Euromonitor]

23 Overview of recent changes in the IP interconnection ecosystem 20 4 Evolution of Internet interconnection As discussed in the previous section, the trends in Internet usage, particularly high bandwidth video traffic, have led to significant increases in traffic. This, in turn, has led to a significant increase in bandwidth capacity demands, which provides a corresponding increase in costs, and concerns over latency, since streamed video content needs to be delivered with minimal delay. At the same time, content is increasingly monetized, providing content providers with a means to help ensure highquality delivery to their users. Here we describe how the main Internet stakeholders have adapted to these new bandwidth demands, how new players have emerged, and how interconnection arrangements have evolved accordingly, changing the traditional hierarchy of the Internet ecosystem in the process. 4.1 Description of the early Internet The Internet ecosystem 15 years ago had a rather simple structure, with relatively clear and hierarchical relationships between the different players: end users and content providers bought Internet access from their local ISP; ISPs bought Internet transit from a limited number of backbone providers to get access to the whole Internet; and backbone providers peered with each other at the US-based NAPs to ensure the full extent of their Internet connectivity. The result was a hierarchy focused largely in the US, with the backbones at the top, interconnecting with one another via peering, and selling the resulting Internet access to downstream ISPs, who in turn sold access to their end users. Figure 4.1 below illustrates the architecture of the early Internet 15 years ago. Annex A provides more background on the relationships between the different players. Content provider & aggregator Backbone 1 All traffic passes through the backbone providers, that carry the traffic between the ISPs End-user ISP 1 ISP 2 NAP Backbone 3 Backbone 2 ISP 3 ISP 4 Internet Access Transit Peering Figure 4.1: Simplified architecture of the early Internet [Source: Analysys Mason]

24 Overview of recent changes in the IP interconnection ecosystem 21 In these years, the online market was still very young and dial-up was the predominant means of Internet access, imposing technical constraints on the services that could be provided. For these reasons, content providers had a limited size. In this context, we should note that it is only 15 years ago that the first of today s major Internet companies started to emerge: Amazon (founded in 1994), Yahoo (1995), ebay (1995), Google (1998), etc. The Internet ecosystem 15 years ago had a rather simple structure characterized by relatively clear and hierarchical relationships, with end users and content providers buying Internet access from ISPs, ISPs buying Internet transit from backbone providers, and backbone providers peering with each other. 4.2 Internet exchange points (IXPs) The nature of Internet exchange has been changing for the past 15 years. When the NSF decided to commercialize the Internet backbone, it designated four dispersed Network Access Points (NAPs), in San Francisco (operated by PacBell), Chicago (BellCore and Ameritech), New York (SprintLink) and Washington DC (MFS). The NAPs were used for public peering, whereby traffic was exchanged using a common switch, which soon congested due to the large volume of traffic. As the historical home of the Internet, a significant amount of international traffic passed through the US at the time, either to connect with US providers, or for transit, sometimes back to the same country of origination. 29 At the same time, the operators of the NAPs had the potential to favor their own services: for instance, when the MAE-East NAP was owned by MCI WorldCom, it required the use of MCI circuits to access MAE-East services. 30 As a result, a migration began towards peering at IXPs around the world. IXPs, such as Equinix, began to open large data centers, where operators could host their servers and/or connect to one another using direct cross-connections. In some countries, non-profit IXPs run by a consortium of members were established today several of the largest IXPs use this model, including LINX in London, and AMS-IX in Amsterdam. The move to IXPs enabled what is known as private peering, whereby service providers do not use a common switch and thus can control congestion over each bilateral link with other service providers. The service providers could also sell or purchase transit over these direct cross-connections at the IXPs. As a result, these IXPs became focal points for the entire Internet ecosystem, where the stakeholders could buy and sell their services to one another at relatively low cost. There are clearly strong network effects that support the growth of an IXP. The more service providers there are in an IXP and the more traffic they exchange, the more attractive it is for other service providers to locate at the same IXP in order to arrange connectivity with the largest number of 29 As all international ISPs at the time had to connect to the US for international transit, they sometimes used these links to exchange domestic traffic, in order to avoid the cost of directly connecting to each other ISP in the country at a time when many domestic telecoms markets were not yet competitive. This flow of traffic out of the country and then back in is often referred to as tromboning. 30 See: Internet Service Providers and Peering, William B. Norton

25 Overview of recent changes in the IP interconnection ecosystem 22 service providers. However, the growth in traffic described above has provided a strong countervailing pressure that has caused IXPs to proliferate across populated areas of countries and regions. Having a dispersion of IXPs has benefits for backbone providers in a peering relationship, in terms of reducing latency, ensuring quality of service (via redundancy) and controlling costs: First, where the two parties in a transmission are in the same region, having a nearby IXP enables the traffic to remain local, rather than tromboning to a more distant location. This enhances the quality of service thanks to reduced latency. A certain level of redundancy is also provided by there being multiple IXPs in different locations in the same region, giving some protection to connectivity in case of natural disaster, for instance. Second, where the two parties in a transmission are in different regions, having dispersed IXPs enables backbone providers to effectively share traffic loads, whereby one provider will carry the originating traffic and the other will carry the return traffic. This second benefit results from what is sometimes referred to as deflection routing or more commonly hot-potato routing, whereby each backbone will exchange the traffic with a peer at the earliest exchange point (for further details see section A.2.3 of the Annex). When the traffic ratio (the ratio of traffic flowing between peers in one direction compared to the traffic flowing in the other direction) is reasonably even, the networks will evenly share the costs associated with carrying traffic exchanged by their users, and thus the agreement is considered fair. As a result, in their publicly available peering policies, backbone providers may require interconnection in up to six different points in the US, for instance (typically with dedicated connections in IXP locations), in order to reduce carriage requirements and balance traffic loads. Backbone providers will also specify a maximum traffic ratio in order to limit the traffic imbalance between them and fairly split the costs of carrying traffic. 31 These types of requirements are designed to reflect comparable costs and benefits, so that the peering arrangement is mutually beneficial to both parties. While IXPs have clearly been beneficial to backbone providers in lowering costs and making more efficient use of capacity, they have also enabled other trends whereby providers rely less on backbones, as discussed below. The development of IXPs has benefited backbone providers by improving the quality of service and reducing traffic carrying costs. However, the benefits rely on roughly even traffic ratios between the peers, as any significant traffic imbalance jeopardizes the peering principle that each party should bear a fair share of the costs of carrying the traffic. 31 See Annex A for an example of a current peering policy.

26 Overview of recent changes in the IP interconnection ecosystem Internet service providers (ISPs) ISPs have reacted to the changing mix of traffic described above in several ways, each of which reduces their reliance on traditional backbones for transit services: The rise in peer-to-peer usage has dramatically increased the traffic that ISPs customers are exchanging between each other (acting in the role of both consumers and suppliers of content). This has led to what is referred to as secondary peering. At the same time, the rapid growth of centralized video content and traffic in sites such as YouTube has led ISPs to connect directly to content providers. This typically results in a form of interconnection known as paid peering. We describe each of these changes in turn. First, the changing mix of traffic led the largest ISPs to reevaluate their need to purchase IP transit from upstream backbone providers. Traditionally, IP transit was used as a means to access traditional content providers, which may be connected to the upstream transit provider or another backbone. On the other hand, IP transit is not necessary to access the customers of another ISP, particularly if that ISP is located in the same IXP. As a result, many ISPs began to peer with one another, as shown in Figure 4.2. Backbone 1 Peering Backbone 2 Figure 4.2: The principle of secondary peering [Source: Analysys Mason] Transit Transit IXP ISP 1 ISP 2 Secondary peering This had two impacts for the ISPs. First of all, it lowered the cost of transit, because the traffic exchanged via secondary peering no longer needed to be sent through the existing transit relationship(s). As transit is based on a measure of traffic volume, this lowered costs. One estimate is that up to 40% of a typical ISP s traffic in the US was peer-to-peer that could be exchanged using secondary peering, with a corresponding reduction of cost. 32 In addition, secondary peering reduced latency for peer-to-peer traffic, as the traffic passed directly between the ISPs rather than via one or 32 The Evolution of the US Internet Peering Ecosystem, William B. Norton, Equinix White Paper 11/19/2003. Norton was the Co-Founder and the Chief Technical Liaison for Equinix. (See

27 Overview of recent changes in the IP interconnection ecosystem 24 more upstream backbones. ISPs that have engaged in such secondary peering are sometimes said to purchase only partial transit from backbones namely only those destinations that they cannot access via peering. Second, the rapid growth of centralized video content and traffic in sites such as YouTube has led to a different form of interconnection known as paid peering. The genesis of paid peering between an ISP and a content provider is similar to that of secondary peering between two ISPs, namely it is an attempt to directly connect in order to reduce transit costs as well as the latency of transmissions. However, there is a fundamental difference resulting from the ratio of traffic involved. When two ISPs peer, it is likely that they will exchange roughly equal amounts of traffic, as their end users are likely to have an equal propensity to generate and consume each other s traffic, be it peer-to-peer or interactive video games or just . On the other hand, when an ISP and a content provider peer, the traffic ratios will not be balanced, as for instance the request for a video consumes very little bandwidth while the video itself takes far more bandwidth: we estimate that the streaming of a threeminute video on YouTube generates around 35 times more downlink traffic than uplink traffic. 33 As a result, the ISP will deliver far less traffic to the content provider than it carries in return. When the imbalance of traffic between service providers rises above a certain traffic ratio, the convention is that the generator of the traffic pays the provider that carries the traffic for the imbalance, in order to cover the cost of capacity needed. Nonetheless, paid peering is advantageous for both parties. Previously, both the ISP and content provider used a transit connection to a backbone to exchange traffic, and each party paid for the traffic that they sent and received. Now, by directly interconnecting, they each lower their transit payments accordingly. Indeed, in this case the content provider would have paid transit fees for the traffic that it delivered to the backbone, while the ISP would have paid similar transit fees for receiving that traffic from the backbone. With paid peering, the content provider pays the ISP directly for delivering the traffic, probably at a rate lower than it would have paid for transit. As a result of these actions on the part of ISPs to lower their transit fees, interconnection has begun to look like Figure 4.3 below. In order to reduce transit fees, ISPs have begun peering with one another, typically at IXP locations for efficiency. At the same time, content providers have begun to interconnect directly with ISPs, using paid peering to compensate for the imbalance of traffic across the connections. This not only further lowers the ISPs transit fees, but also results in fees from the paid peering. These new interconnections come at the expense of backbones, which, as a result, end up selling less transit to the ISPs and content providers. 33 The video streaming request sent by a user to YouTube generates around 430 kbytes of uplink traffic, while the resulting video stream generates around 15 Mbytes.

28 Overview of recent changes in the IP interconnection ecosystem 25 Content provider & aggregator Backbone 1 ISP 2 End-user Backbone 3 Backbone 2 IXP ISP ISP 4 Single entity ISP 3 Internet Access Transit Partial transit Peering Secondary peering Paid peering Figure 4.3: Interconnection possibilities that emerged in the past 15 years [Source: Analysys Mason] Many route around options have emerged for ISPs and content providers to carry Internet traffic while avoiding Internet backbone transit costs. These options include secondary peering between ISPs and partial transit agreements. Traffic imbalances have also led to the development of paid peering, which is advantageous for ISPs and content providers. 4.4 Content providers Paid peering with ISPs is only one of the means that content providers are using to lower their costs efforts that go hand in hand with ways to reduce latency and thereby improve the experience for their users. Another technique used for this purpose is called web caching. A web cache used by an ISP stores a copy of a web page or video requested by an end user. The ISP can then use the cache to satisfy subsequent requests for the same content, instead of reloading it from the original source. In other words, if 500 users view the same popular video, it can be downloaded once from the original content provider over the transit connection, and the next 499 times it will be viewed from the web cache within the ISP s network, significantly lowering the amount of traffic over the transit connection, and therefore reducing transit payments. Web caching may be used by ISPs, as described above, but this technique is more typically used either by third-party content delivery networks (CDNs) or by larger content providers, which are effectively building their own delivery networks in order to deliver content closer to (or from within) the ISPs networks. Commercial CDNs effectively compete with backbones in order to deliver their clients content directly to ISPs. CDNs ensure that the content of their clients is distributed to end users with good performance, independently of the location of the end user. To fulfill this role, a CDN typically owns a network of caching servers, connected by fiber, which feed copies of content to ISPs

29 Overview of recent changes in the IP interconnection ecosystem 26 at a limited number of points of presence (PoPs) where networks interconnect. 34 The largest CDNs include Akamai (founded in 1998) and Limelight (2001). According to one estimate, CDNs deliver close to 10% of all Internet traffic. 35 See Figure 4.4 below for a stylized diagram of how a CDN interacts with the other players in the Internet ecosystem. The use of CDNs to carry content traffic effectively reduces the traffic to be carried by the backbone provider, and avoids any traffic imbalance issues. However, at the same time it reduces the transit revenues of backbone providers. Regional ISP Content Delivery Network Mobile operator Regional ISP Backbone network A Backbone network B Content provider Aggregation & Access Backbone Aggregation & Access End-user Figure 4.4: Structure of a content delivery network [Source: Analysys Mason] Content providers themselves may use a similar configuration to build a content delivery network for their own content. For instance, Google has embarked on a Google Global Cache project with similar goals to a CDN, in this case to locate Google servers within ISP networks. 36 The difference from a CDN is that Google servers are dedicated to Google content. Other content providers such as Microsoft are also building such networks to deliver content, such as Microsoft s video games. The difference in the traffic carried over the backbone can be significant: Arbor Networks estimates that Google traffic represents at least 6% of global Internet traffic, and more than 60% of that traffic is delivered to ISPs over direct connections rather than via third-party providers. 37 Figure 4.5 shows how the percentage of Google traffic using private peering has increased in recent years. 34 CDNs usually try to connect directly to ISPs for cost and performance reasons, but may need in some instances to buy transit from backbone providers (when they do not own the fiber network), and also buy transit from ISPs (when they cannot get free peering from large ISPs). 35 ATLAS Internet Observatory, 2009 Annual Report, Arbor Networks, Inc., University of Michigan, Merit Network, Inc., Pre-Publication Draft, page See 37 Source: How Big is Google?, Craig Labovitz,

30 Percentage Apr-08 Jun-08 Aug-08 Oct-08 Dec-08 Feb-09 Apr-09 Jun-09. Aug-09 Oct-09 Dec-09 Feb-10 Overview of recent changes in the IP interconnection ecosystem % 90% 80% Figure 4.5: Percentage of Google traffic using direct peering [Source: Arbor Networks 38 ] 70% 60% 50% 40% 30% 20% 10% 0% Even smaller content providers can also use technology to lower their costs, avoiding not just transit charges, but also the use of CDNs. For instance, Vudu delivers high-quality video on demand to end users TVs via a dedicated set-top box that is external or integrated into a TV or Blu-ray player. Each device has a large hard drive, which contains the first few seconds of each title in order to enable the title to begin as soon as it is chosen. The rest of the chosen content is pieced together via an Internet connection to the device, using peer-to-peer networking from other Vudu devices that have stored that content. As a result, a user is able to watch high-definition movies with no delay with as little as a 4Mbit/s broadband connection. In this way, Vudu uses its customers own broadband connections to deliver content to other users in lieu of a more expensive CDN or backbone service. As a result of the volume of traffic that they deliver, and the associated cost of delivering that traffic, content providers have a strong incentive to build out their networks in order to reduce their transit requirements and deliver the traffic directly to the ISP network, if not from within the ISP s network. As a result of the increasing monetization of content, providers also have the means to engage in such network build-outs. But even if CDNs are increasingly used or built to deliver Internet content to ISPs, in particular to deliver the growing share of video traffic for which caching is appropriate, backbones and/or ISPs still provide significant infrastructure to deliver the content. As explained earlier, CDNs typically distribute their caching servers into specific PoPs to connect with backbones and ISPs networks. The selection of these points by the CDNs is optimized in terms of coverage, i.e. CDNs will primarily set up PoPs in regions concentrating significant amounts of Internet traffic. However, backbones and ISPs then ensure the delivery of content to the end users from these PoPs to the end users, regardless of final location. 38

31 Overview of recent changes in the IP interconnection ecosystem 28 Overall, the traffic carriage effort supported by backbones and ISPs is significant in comparison with the CDNs. CDNs typically connect at tens of PoPs: for instance, Limelight notes that in 2010 its network consisted of 75 PoPs where it interconnected globally, in order to deliver traffic to more than 900 ISPs and backbones. This is broadly similar to the number of PoPs that each single major backbones typically possesses. 39 However, the backbones and/or ISPs carry the Internet traffic further into the network, in order to reach the central offices 40 counted in the US (as of February 2010), from where it is carried over the last mile to end users for DSL services. 41 On average, each PoP may connect to up to 300 or more central offices, requiring significant network resources to carry the content from the CDN to the last-mile connection. In conclusion, as explained in the previous section, there are now many situations where content providers can bypass the backbone providers transit fees. Content providers also have the opportunity to rely on their own private network or a third-party CDN to carry their traffic to the ISP. Either way, the consequence has been the continuing trend of providers relying less on backbones, which may now be avoided for a significant amount of traffic between content providers and end users, even if they keep a key role in delivering content to and from end-user networks. Content providers can employ many different options to carry their Internet traffic to the ISP (e.g. using CDNs or building their own private networks), and can leverage their expanding market power and financial resources to rely less on backbone providers. 4.5 Impact on backbone providers From the backbone provider s point of view, these market evolutions have had impacts at two levels: First, changes in traffic patterns have led to changes in the arrangements between backbone providers and their traditional customers, the ISPs and content providers. These changes have impacted the traditional hierarchy of the Internet ecosystem and put downward pressure on transit prices. Second, although many traditional backbone customers are relying less on backbones to accommodate changing traffic patterns, the same trends are impacting the traffic that they still deliver to the backbones as customers, which puts pressure on the relationship between backbone 39 The number of PoPs on backbones is usually not publicly revealed. However, studies providing solid estimates suggest that large backbones in the US typically own around 50 to 100 PoPs (See 40 Central offices are network facilities where end users lines get locally connected to the core network (local loop) for xdsl services. To deliver the traffic to end users, ISPs carry aggregated Internet traffic to the central offices, where it is then distributed to the end users over the last portion of the network (last mile). 41 Source: FCC, Trends in Telephone Service, FCC, September 2010 ( Similar arrangements are true for cable networks.

32 Overview of recent changes in the IP interconnection ecosystem 29 providers. In particular, the increase in content traffic being delivered to end users is impacting the conditions under which backbones agree to peer with one another. With respect to the first issue, the changes impacting ISPs and content providers as described in the previous sections lead to a reduction in demand for transit. This is due to the changes with regards to backbones customers at both ends of the value chain: namely ISPs and content providers, which are able to self-supply a growing portion of their needs, and which are also entering into multiple agreements with peers and other suppliers. This is true not just at a national level, but also internationally as more and more traffic is regionalized through large IXPs that reduce a reliance on international transit for traffic exchange. As a result, this puts pressure on transit prices (shown below) as backbones effectively face greater competition from the self-supply of their customers. Further, the changing nature of Internet traffic also impacts the relationships between the backbone providers themselves. In particular, with the increase of bandwidth-intensive content, particularly video, comes an increase in an imbalance of traffic sent from the content providers to the customers of the ISPs. As a result, without a change in the mix of customers, backbones with content providers as customers will send more traffic to backbones with ISP customers. As noted in Section 4.2, peering policies typically specify a maximum traffic ratio governing the traffic flow between peering partners; this has been a feature of peering policies at least since backbones began making their peering policies publicly available a decade ago. 42 Increases in traffic ratios resulting from delivering content can begin to overrun the traffic ratio requirements in the corresponding peering policies. With the default hot-potato routing that is common for peering arrangements, the sender of the traffic quickly hands over the traffic to the receiving network for the latter to carry across the network to the recipient, as described below in Annex A.2.3. As a result, a high traffic ratio can impose costs on the receiving network, as it carries far more traffic than it hands off to the other network in return. Specifically, the backbone providing the content will quickly hand it to the backbone with ISP customers that are receiving the content. As a result, the latter backbone has greater network costs in transporting the incoming traffic from the backbone providing the content, which could be seen as a cost subsidy from the former to the latter. As these costs result from content being exchanged near to the source under hot-potato routing, one solution is what is known as cold-potato, or best-exit routing, in which the sending network keeps the high-bandwidth traffic on its own network longer before exchanging it closer to the consumers. However, given the ratio between PoPs and the number of distribution points in an ISP s network, cold-potato routing may still not alleviate the imbalance of costs, implying that the ISP may subsidize the delivery of traffic from the content provider, on behalf of its customers who may already be paying for the relevant content (directly or via associated advertising). 42 For instance, in January 2001 WorldCom introduced a publicly-available peering policy specifying the requirements that a network must meet in order to peer with World Com. This policy included a clause with the following requirement: The ratio of the aggregate amount of traffic exchanged between the Requester and the WorldCom Internet Network with which it seeks to interconnect shall be roughly balanced and shall not exceed 1.5:1. See Peering_Policy_files/UUNET%20Global%20Site%20WorldCom% 20Policy%20for%20Settlement-Free%20Interconnection%20with%20Internet%20Networks.htm

33 USD per Mbps. Overview of recent changes in the IP interconnection ecosystem 30 Other solutions to the imbalance of costs issue include paid peering, discussed above, or even moving to a transit relationship whereby the backbone whose traffic causes the imbalance becomes a customer in order to cover the costs of carrying the traffic. Any such fees are kept in check by the ability of the content provider to use a CDN or build its own network to deliver the content closer to the ISP for delivery. However, such changes in peering arrangements can be contentious, as seen in the dispute between Level 3 and Comcast in late The changes described above have put downward pressure on Internet transit prices charged by backbone providers. The result of the multiplicity of possible interconnections also encouraged the consolidation of the Internet ecosystem: ISPs and content providers are in a position to bypass the backbone providers, leading to an increase in the level of competition between backbones for Internet transit services. Further, backbones must compete on the price of transit in order to generate the traffic volume to continue to meet peering requirements. This increased competition can be observed through the continuous decline of transit prices over the past five years (Figure 4.6 below), with no sign of respite. While transit prices have fallen as a result of these changes, there has been no reduction in the absolute amount of traffic carried by backbones, which continue to supply new capacity to carry the explosion of video traffic, albeit at lower and lower absolute prices Figure 4.6: Median global transit prices per Mbit/s, Gigabit Ethernet [Source: Telegeography] Chicago Houston Los Angeles Miami New York San Francisco 43 While the particulars of the dispute between Level3 and Comcast are covered by confidentiality agreements, after Level3 signed a content distribution agreement with Netflix to deliver online movies, it appears that Level3 began to send up to five times more traffic to Comcast than it received in return. As this seems to have been outside the ratio agreed in their peering policy, Comcast asked Level3 to sign a paid peering arrangement that Level3 perceived to be unfair. A further description of the dispute can be found in

34 Overview of recent changes in the IP interconnection ecosystem 31 The result of the multiplicity of possible interconnections has encouraged the consolidation of the Internet ecosystem: ISPs and content providers are in a position to bypass the backbone providers, leading to an increase in the level of competition between backbones for Internet transit services. The result has put downward pressure on Internet transit prices charged by backbone providers, while the traffic demands continue to soar.

35 Overview of recent changes in the IP interconnection ecosystem 32 5 Conclusion The Internet ecosystem has changed dramatically since the commercialization of the Internet backbone 15 years ago. From a rather simple ecosystem with relatively clear and hierarchical relationships, the main players (end users, content providers, ISPs and backbone providers) have steadily adapted to changing market conditions, accommodating the increase in the demand and willingness to pay for Internet services and corresponding increases in traffic. They have also engaged in vertical consolidation, as content providers started to build their own networks and ISPs began to bypass the backbone networks. In particular, these players have altered the nature of their relationships, where necessary, in order to achieve enhanced connectivity and reduce their transit costs: The increase in utilization of IXPs around the world facilitated distributed interconnection between peers and transit customers, but also helped in the efforts to reduce transit costs. The development of direct interconnections between ISPs and other ISPs (secondary peering) or content providers (typically with paid peering) also helped to reduce transit costs. The building of private networks by content providers or third-party CDNs to deliver content to ISPs has further reduced transit costs and the resulting revenues for backbones. In the same way that competing Internet backbones have developed peering arrangements as an input to the transit services that they sold to ISPs and content providers, the ISPs and content providers themselves have developed new arrangements in response to the changes in content traffic, notably the steep rise in video, as well as the rise of distribution models such as peer-to-peer. The main attribute that has characterized Internet interconnection arrangements is a flexible system within which players are able to adapt to changes in their own circumstances in a way that also furthers the mutual interests of the entire ecosystem. The commercially-driven evolution of Internet interconnection stands in contrast to the regulation that governs interconnection of telecommunications services, which may share the same network infrastructure with the Internet, and involve many of the same players. For example, in the past decade, during which the evolution in the Internet described above took place, the FCC has made countless attempts to modify the inter-carrier compensation system in response to changes in telecommunications a process that is still ongoing. 44 In contrast, since the commercialization of the Internet backbone, the Internet ecosystem has long proven itself to be able to develop and sustain interconnection in the absence of sector-specific regulation and it now has shown itself to also be able to adapt well to rapid and profound market changes without regulatory intervention. 44 For a list of various ongoing FCC proceedings on inter-carrier compensation reform, see

36 Overview of recent changes in the IP interconnection ecosystem B-1 Annex A: Introduction to the IP interconnection ecosystem In this annex, we describe the different players of the Internet ecosystem, as well as the traditional relationships between them beginning in the early Internet 15 years ago, including the peering and transit relationships. A.1 Description of the various players in the Internet ecosystem The Internet is a network of networks interconnected with one another, and operated by different Internet providers. Traditionally, we can distinguish four different roles in the value chain of content delivery over the Internet (involving data transfers between content providers and the end users over an Internet connection): content providers and aggregators internet service providers (ISPs) internet backbone providers end users. This classification can still be used to describe the Internet of today, and thus will be used in the rest of this paper, although new players have emerged and others now cover more than one role. These new players are described in Section 4 of this document. The figure below provides a simplified illustration of the value chain of content delivery over the Internet: Content provider/ aggregator ISP (Internet access) Internet backbone providers ISP (Internet access) End user ISP 1 Backbone Backbone ISP 2 Network A Network B Settlementfree transit, paid transit, or peering Figure A.1: Value chain of content delivery over the Internet, in the Early Internet [Source: Analysys Mason] As elements in this value chain, the four types of players listed above could be defined as follows: A content provider (and aggregator) creates content for the Internet and/or aggregates this content, to make it available to its Internet users or customers. A content provider will get Internet access via an Internet service provider, which provides the data transmission service to the rest of the Internet. The roles of content creation and content aggregation may belong to different entities, and for instance content aggregators may gather content from multiple

37 Overview of recent changes in the IP interconnection ecosystem A-2 external sources. However, for clarity purposes, we will consider that both roles are played by a single entity, for example The New York Times online. An ISP offers its customers access to the Internet via a data transmission technology such as dial-up, DSL, cable modem, wireless or dedicated high-speed connection. Its customers are Internet end users and content providers. In this paper, we will consider that ISPs have either a regional or national footprint, for example, America Online An Internet backbone provider delivers traffic to and from third party networks through its infrastructure of national or international high-speed fiber optic networks. An Internet backbone provider network is typically connected to several ISP networks and other backbone providers networks, and may also act as an ISP in selling service directly to large enterprise users, for example, Level 3 An end user accesses the Internet via a fixed or mobile device connected to the Internet by an ISP. This could be an individual consumer or an enterprise. A.2 Identification of the relationships between the various stakeholders The financial relationships between the different players within the Internet ecosystem are summarized in Figure A.2 and described at a high level below. Content provider/ aggregator ISP Internet backbone provider Internet backbone provider and ISP End user Money flow Figure A.2: Money flows between the players in the Internet ecosystem [Source: Analysys Mason] A.2.1 Relationship between content provider and ISP: Internet access Content providers traditionally did not possess their own network infrastructure, and therefore bought Internet access from an ISP: the ISP allows the content provider make its content available to any Internet user, by ensuring the routing of traffic between the content provider s servers and the Internet end users. When the content provider is a sizable organization, it connects to the ISP with high-capacity links, dedicated routers and specific servers. When the content provider is a smaller organization or an individual (e.g. blog owner), the ISP generally hosts the content on its own servers and will

38 Overview of recent changes in the IP interconnection ecosystem A-3 not use dedicated infrastructure. It should be noted that, depending on the number of users that access its content, the quantity of traffic generated, and the network infrastructure owned and operated by the content provider, the content provider can either connect to an ISP or directly to a backbone provider effectively acting as a large ISP. In terms of money flows, the content provider typically pays the ISP for this service, based on the volume of the interconnection between them. A.2.2 Relationship between ISP and Internet backbone: Internet transit An ISP s footprint may be limited to a certain geographical area, and an ISP s network consists of its own users. In order to offer its client a global reach to the Internet, ISPs enter into transit relationships with backbone providers, who can provide the ISP with access to the rest of its transit customers as well the customers of other backbones with which it peers (as described below). In the simplest case, an ISP establishes a single connection to a backbone provider at a PoP. In reality, an ISP often has more than one PoP at a national level, and thus several separate connections to a backbone provider at those PoPs. An ISP may also be a customer of multiple backbone providers, often referred to as multi-homing, and have connections to each one of them at one or more PoPs. A.2.3 Relationship between Internet backbones: Internet peering A backbone provider s network consists of its own customers in order to sell access to the entire Internet, it must connect to other backbone providers. Backbone providers of a similar size will typically enter in a relationship called peering, whereby the backbones exchange traffic between their customers. Peering is typically settlement-free, meaning that neither backbone provider pays the other for the exchanged traffic. Instead, peering acts as an input that enables backbones to sell access to the entire Internet to its own transit customers such as ISPs. Therefore, backbone providers will only enter into a peering relationship with another backbone when it is mutually beneficial. For this reason, backbone providers will enter into peering agreements only with peers that meet a list of criteria. These criteria stem in large part from the decisions that peering partners take in how to route traffic between them, in response to two features of IP interconnection. First, a central feature of the Internet Protocol is that it is connection-less, meaning that the transmission path is determined by network availability and not a pre-determined circuit, and in particular, the path that a web request takes need not be the same as the return path taken by the response. Second, Internet

39 Overview of recent changes in the IP interconnection ecosystem A-4 backbones will typically interconnect in multiple geographical locations for efficiency, in order to keep local traffic in the same region, and for redundancy to ensure a continual connection if one link fails. As a result, two backbones that are connected in a number of places must determine where traffic is exchanged between them in order to fairly distribute the carrying of exchanged traffic. The common solution is what is known as hot-potato routing, in which, as the name suggests, the originating backbone hands off the packets as close as possible to the point of origination, and likewise the peering partner will do the same with the return traffic. A diagram of this is shown below in Figure A.3. 4 End user Backbone/ISP 1 1 IXP 1 IXP 2 3 Backbone/ISP 2 Content provider 2 Figure A.3: Hot-potato routing [Source: Analysys Mason]

40 Overview of recent changes in the IP interconnection ecosystem A-5 In this example, an end user connected via an ISP to Backbone 1 is requesting content from a content provider connected via another ISP to Backbone 2. The following steps occur: 1. Backbone 1 exchanges the request with Backbone 2 at the nearest IXP, which is IXP This traffic is then carried by Backbone 2 to the content provider. 3. The return traffic is then transferred back to Backbone 1 through the nearest IXP (IXP 2). 4. Finally, Backbone 1 carries the return traffic to the end user Generally speaking, peering policies have two relevant sections, aimed in large part at managing the traffic flow between the networks: Network requirements. This refers to the geographical scope of the network and the raw capacity of the network. These requirements are included in peering policies to ensure that peering partners have made similar investments and that the backbone requesting peering will not be able to free ride on investments by the backbone receiving the request. For instance, a national backbone will not want to peer with a regional backbone, because in this case all traffic exchange would have to take place within the regional backbone s reach, and the national backbone would carry traffic outside the regional backbone s network in both directions. Peering requirements. In addition, there may be geographical and traffic requirements specific to the peering connections between the two backbones. Again, this is to prevent free riding and ensure that both backbones receive equal benefits from the connections. For instance, the geographical requirements ensure that connections take place in multiple locations across the country in order to distribute traffic exchange. Further, some backbones require that the traffic that they would receive from the backbone requesting peering is roughly balanced with the traffic that they would send to that backbone, so that with hot-potato routing they do not use more capacity for the peering connection than the other backbone uses. As an example of imbalanced traffic, with reference to Figure A.3 above, Backbone 2, with the content provider, would carry the relatively small requests for web content across most of its backbone, in step (2) of the diagram, while Backbone 1, with the end user, would carry all of the web content back across its network, as per step (4). As a result, Backbone 1 would have to devote more capacity to the peering relationship than Backbone 2, whose customer is generating the bulk of the exchanged traffic in the form of the web content. As expressed in many, if not most, peering relationships, networks in the position of Backbone 1 would not consider this relationship to be mutually beneficial unless there were offsetting sources of traffic to keep the traffic in ratio.

41 Overview of recent changes in the IP interconnection ecosystem A-6 As an example, the list below summarizes the main requirements for a carrier to peer with AT&T 45 Type of requirement Capacity National presence International presence Traffic Bandwidth Exclusivity Network Operations Management Traffic ratio Routing policy Financial stability Detailed requirement A peer must operate a US-wide IP backbone whose links capacity are primarily 10 Gbit/s or greater A peer must meet AT&T at a minimum of three mutually agreeable geographically diverse points in the US. The US interconnection points must include at least one city on the US east coast, one in the central region, and one on the US west coast, and must currently be chosen from AT&T peering points in a list of selected metropolitan areas (New York City, Washington DC, etc.) A peer must interconnect in two mutual non-us peering locations on distinct continents where peer has a non-trivial backbone network. These non-us peerings will be with AT&T s regional AS only Peer s traffic to/from the interconnected AT&T US network must be on-net only and must amount to an average of at least 7 Gbps in the dominant direction to/from AT&T in the US during the busiest hour of the month Interconnection bandwidth for private peers must be at least 1 Gbit/s at each US interconnection point A network (ASN) that is a customer of an AT&T US network for any dedicated IP services may not simultaneously be a settlement-free peer of that same network Peer must have a professionally managed 247 Network Operation Center. Peer must agree to repair or otherwise remedy any problems within a reasonable timeframe. Peer must also agree to actively cooperate to resolve security incidents, denial of service attacks, and other operational problems Peer must maintain a balanced traffic ratio between its network and AT&T, and must have: No more than a 2:1 in-out ratio of traffic (traffic into AT&T - traffic out of AT&T network), on average each month A reasonably low peak-to-average ratio Existing peers whose in-out ratio rises above 2:1 will be expected to work with AT&T to implement best-exit routing or to take other suitable actions to balance transport costs Peer must abide by the following routing policy: Peer must use the same peering AS at each US interconnection point and must announce a consistent set of routes at each point, unless otherwise mutually agreed No transit or third party routes are to be announced; all routes exchanged must be peer's and peer s customers' routes Peer must filter route announcements from its customers by prefix Neither party shall abuse the peering relationship by engaging in activities such as: pointing a default route at the other or otherwise forwarding traffic for destinations not explicitly advertised, resetting next-hop, selling or giving next-hop to others Peer must be financially stable Figure A.4: AT&T peering requirements [Source: AT&T 46 ] 45 Note: these are the conditions to peer with AT&T Worldnet network.

42 Overview of recent changes in the IP interconnection ecosystem A-7 In contrast to a transit connection, which typically only requires a connection at one place and gives access to the entire Internet, peering requires multiple connections, and gives access only to the peer s customers. Therefore, backbone providers must connect to multiple other backbone providers in order to achieve a global reach. While peering is traditionally settlement-free, another form paid peering also exists. Paid peering is a mix of peering and transit; similar to peering, it only provides access to a peer s direct customer base, and not the entire Internet, but similar to transit one provider pays the other for this access (See section 4 for more details). A.2.4 Relationship between ISP and end user: Internet Access The ISP provides the end user with Internet connectivity for a certain fee; this is typically a flat fee per month, but the fee may also depend on the volume of consumption. A.3 Evolution of interconnection arrangements It is also important to note that in this ecosystem, backbone providers had a fairly clear evolution path. In particular, smaller backbone providers purchase transit from larger backbone providers, as they could not meet the requirements to peer with them. But in this dynamic system, backbone providers that grew naturally have gradually changed their relationships with other providers, as they progressively met the requirements to peer with larger backbone providers and eventually grew to the point where they could begin to sell transit themselves. Figure A.5 below illustrates this evolution of backbone providers. 46