Interconnection 101 KEY FINDINGS

Transcription

1 Interconnection 101 As cloud usage takes off, data production grows exponentially, content pushes closer to the edge, and end users demand data and applications at all hours from all locations, the ability to connect with a wide variety of players becomes ever more important. This report introduces interconnection, its key players and business models, and trends that could affect interconnection going forward. KEY FINDINGS Network-dense, interconnection-oriented facilities are not easy to replicate and are typically able to charge higher prices for colocation, as well as charging for cross-connects and, in some cases, access to public Internet exchange platforms and cloud platforms. Competition is increasing, however, and competitors are starting the long process of creating network-dense sites. At the same time, these sites are valuable and are being acquired, so the sector is consolidating. Having facilities in multiple markets does seem to provide some competitive advantage, particularly if the facilities are similar in look and feel and customers can monitor them all from a single portal and have them on the same contract. Mobility, the Internet of Things, services such as SaaS and IaaS (cloud), and content delivery all depend on network performance. In many cases, a key way to improve network performance is to push content, processing and peering closer to the edge of the Internet. This is likely to drive demand for facilities in smaller markets that offer interconnection options. We also see these trends continuing to drive demand for interconnection facilities in the larger markets as well. AUG 2015

2 i 451 RESEARCH ABOUT 451 RESEARCH 451 Research is a preeminent information technology research and advisory company. With a core focus on technology innovation and market disruption, we provide essential insight for leaders of the digital economy. More than 100 analysts and consultants deliver that insight via syndicated research, advisory services and live events to over 1,000 client organizations in North America, Europe and around the world. Founded in 2000 and headquartered in New York, 451 Research is a division of The 451 Group Research, LLC and/or its Affiliates. All Rights Reserved. Reproduction and distribution of this publication, in whole or in part, in any form without prior written permission is forbidden. The terms of use regarding distribution, both internally and externally, shall be governed by the terms laid out in your Service Agreement with 451 Research and/or its Affiliates. The information contained herein has been obtained from sources believed to be reliable. 451 Research disclaims all warranties as to the accuracy, completeness or adequacy of such information. Although 451 Research may discuss legal issues related to the information technology business, 451 Research does not provide legal advice or services and their research should not be construed or used as such. 451 Research shall have no liability for errors, omissions or inadequacies in the information contained herein or for interpretations thereof. The reader assumes sole responsibility for the selection of these materials to achieve its intended results. The opinions expressed herein are subject to change without notice. New York 20 West 37th Street, 6th Floor New York, NY Phone: Fax: San Francisco 140 Geary Street, 9th Floor San Francisco, CA Phone: Fax: London Paxton House (5th floor), 30 Artillery Lane London, E1 7LS, UK Phone: +44 (0) Fax: +44 (0) Boston 1 Liberty Square, 5th Floor Boston, MA Phone: Fax:

3 ii Interconnection 101 TABLE OF CONTENTS SECTION 1: EXECUTIVE SUMMARY INTRODUCTION KEY FINDINGS METHODOLOGY SECTION 2: WHAT IS INTERCONNECTION, AND WHERE DOES IT COME FROM? CARRIER-NEUTRAL DATACENTER VS MEET-ME ROOM Figure 1: Carrier-Neutral Datacenter Compared with Meet-Me Room INTERCONNECTING THE INTERNET Private Interconnection Figure 2: Internet Transit Figure 3: Private Peering Figure 4: Internet Transit Plus Peering Public Interconnection or Public Peering Figure 5: Public Peering Platform Figure 6: Public Peering in the US vs. Europe SECTION 3: INTERCONNECTION AS A BUSINESS COMPONENTS The Building Bandwidth Cross-Connects Public Peering Platform Access to Other Customers in the Facility, Particularly Cloud Providers Additional Services SUPPLY AND DEMAND Supply Demand CUSTOMERS Figure 7: Customers of Interconnection Facilities Figure 8: Drivers of Facility Selection

4 iii 451 RESEARCH SECTION 4: INTERCONNECTION PROVIDERS 18 Figure 9: 451 Research Interconnect Market Map TM Figure 10: Interconnection Provider Segments Figure 11: Summary Chart: Market Challenges and Innovations SECTION 5: EVOLUTION OF INTERCONNECTION: TRENDS AND DISRUPTORS CONTINUED GROWTH OF INTERNET TRAFFIC AND THE NEED FOR INTERCONNECTION INCREASE IN THE NUMBER OF FIRMS INTERCONNECTING GROWING REQUIREMENT FOR INTERNET CONNECTIVITY AT THE EDGE CLOUD S IMPACT ON INTERCONNECTION NET NEUTRALITY PRIVATIZATION OF THE INTERNET COMPETITIVE CHANGES Open-IX European Exchanges in the US Additional Competition TECHNOLOGY TRENDS SECTION 6: THE 451 TAKE 30 APPENDIX A: GLOSSARY 31 APPENDIX B: KEY CARRIER HOTELS IN NORTH AMERICAN MARKETS 33 APPENDIX C: LOCATIONS FOR DIRECT CONNECTIONS TO CLOUD PROVIDERS 34 AWS Direct Connect Locations Microsoft Azure ExpressRoute Locations APPENDIX D: OPEN-IX CERTIFIED PROVIDERS 35 INDEX OF COMPANIES 36

5 1 451 RESEARCH SECTION 1 Executive Summary 1.1 INTRODUCTION Interconnection has come a long way since telecommunications providers connected their networks in order to exchange voice traffic. Now, in addition to carriers, many other kinds of firms need to connect with each other to exchange data traffic, and interconnection itself has become a business. Facilities where the largest number of firms can meet have become extremely valuable. This report looks at the business of interconnection and discusses trends that are likely to impact it going forward. 1.2 KEY FINDINGS Network-dense, interconnection-oriented facilities are not easy to replicate and are typically able to charge higher prices for colocation, as well as charging for cross-connects and, in some cases, access to public Internet exchange platforms and cloud platforms. Competition is increasing, however, and competitors are starting the long process of creating network-dense sites. At the same time, these sites are valuable and are being acquired, so the sector is consolidating. Having facilities in multiple markets does seem to provide some competitive advantage, particularly if the facilities are similar in look and feel and customers can monitor them all from a single portal and have them on the same contract. Mobility, the Internet of Things, services such as SaaS and IaaS (cloud), and content delivery all depend on network performance. In many cases, a key way to improve network performance is to push content, processing and peering closer to the edge of the Internet. This is likely to drive demand for facilities in smaller markets that offer interconnection options. We also see these trends continuing to drive demand for interconnection facilities in the larger markets as well. 1.3 METHODOLOGY This report on interconnection services is based on a series of in-depth interviews with a variety of stakeholders in the industry, including technology vendors, surveys and interviews of IT managers at end-user organizations across multiple sectors, datacenter service providers and providers of connectivity services. This research was supplemented by additional primary research, including attendance at trade shows and industry events. Please note that the names of vendors and service providers are meant to serve as illustrative examples of trends and competitive strategies; company lists are comprehensive, but are not intended to be exhaustive. The inclusion (or absence) of a company name in the report does not necessarily constitute endorsement.

6 2 Interconnection 101 Reports such as this one represent a holistic perspective on key emerging markets in the enterprise IT space. These markets evolve quickly, so 451 Research offers additional services that provide critical marketplace updates. These updated reports and perspectives are presented on a daily basis via the company s core intelligence service, 451 Research Market Insight. Forward-looking M&A analysis and perspectives on strategic acquisitions and the liquidity environment for technology companies are also updated regularly via 451 Market Insight, which is backed by the industry-leading 451 Research M&A KnowledgeBase. Emerging technologies and markets are also covered in additional 451 Research channels, including Datacenter Technology; Enterprise Storage; Systems and Systems Management; Enterprise Networking; Enterprise Security; Data Platforms & Analytics; Dev, Devops & Middleware; Business Aps (Social Business); Managed Services and Hosting; Cloud Services; MTDC; Enterprise Mobility; and Mobile Telecom. Beyond that, 451 Research has a robust set of quantitative insights covered in products such as ChangeWave, TheInfoPro, Market Monitor, the M&A KnowledgeBase and the Datacenter KnowledgeBase. All of these 451 Research services, which are accessible via the Web, provide critical and timely analysis specifically focused on the business of enterprise IT innovation. This report was written by Jim Davis, Senior Analyst, Service Providers, and Kelly Morgan, Research Director, Datacenters. Any questions about the methodology should be addressed to Jim Davis or Kelly Morgan at: jim.davis.@451research.com or kelly.morgan@451research.com. For more information about 451 Research, please go to:

7 3 451 RESEARCH SECTION 2 What Is Interconnection, and Where Does It Come From? The very essence of the Internet is interconnection; the word is a shortened version of internetworking, because the Internet is a system of millions of networks that have been linked together by the use of standard protocols for communication. Beyond the technical standards, however, interconnection has become a business in its own right. In this report, we focus on interconnection services, key players and business models particularly within and between datacenters. Many interconnect locations got their start as carrier hotels. National telecom providers have always needed to hand off international traffic to carriers in other countries. They connected with each other at key locations to make this handoff, often near the landing points of undersea cables. As national carriers have been deregulated and competition within the US has grown, competing carriers have had to connect their networks to exchange national as well as international traffic. As a result, the number of carrier hotels and the locations where they are needed have multiplied. Due to the concentration of carriers, these carrier hotels have also become key locations for Internet connectivity. The original buildings where carriers connected their networks belonged to the carriers themselves, to the incumbents and/or the long-haul network providers. These tended to be central offices (COs), where the owner had telco equipment but leased out extra space to other carriers. Often, the owner provided the only means of network connectivity to the facility. However, there was not necessarily much incentive for the carrier-owner to maintain, expand or upgrade the CO to add capacity for potential competitors. Local carriers sought locations that were more neutral. These were often office buildings in the center of cities, to which several providers already had fiber connectivity. The carriers paid rent to the building owner and the connections were made in a central location in the building that came to be called the meet-me room. Facilities where participants had multiple network options to access the building became known as carrier-neutral. The facilities usually are not owned by carriers, but sometimes can be if the carrier offers interconnection without requiring that participants use its network. For example, NAP of the Americas in Miami is a carrier-neutral facility owned by Verizon. Some carrier hotels grew up after market deregulation; in the US, One Wilshire s status as a carrier hotel began with then-regional telco PacBell refusing to allow competing telecom service provider MCI (which at the time was focused on long-distance calling) to ban competitors switches and circuits inside its central switching facility at 400 South Grand in Los Angeles. MCI chose a building nearby that had a sightline for its microwave transmission equipment. Over time, other telecom providers began bringing fiber into the building, eventually turning it into one of the most interconnected hubs for Internet and telecom services in the world. Similar examples can be found in Europe. In Frankfurt, datacenter and IT services provider ITENOS started by building out a former bakery for a telecom client in 1995 and over the next decade adding space for carriers in several nearby buildings, including Kleyerstrasse 90. Kleyer 90 s list of carrier tenants meant it was considered a carrier hotel by the time Equinix acquired it in 2013.

8 4 Interconnection 101 Other carrier hotels, such as 60 Hudson Street in New York City, had a longer historical link to network interconnection. The building was originally the headquarters of the Western Union Company, the provider of telegraph communication services founded in The building served as a point of connection for the firm s telegraph network; now the building houses more than 100 companies from around the globe that interconnect at the building s meet-me room. 2.1 CARRIER-NEUTRAL DATACENTER VS MEET-ME ROOM In original carrier hotels, the meet-me room was where the physical interconnections were made. Now, however, the term carrier-neutral datacenter may be used to describe an interconnection location. Figure 1 notes some of the differences between the two, but there can also be some overlap between the terms. For example, a Telx facility within a larger building can be considered a carrier-neutral datacenter on its own and can also be the building s meet-me room. Perhaps the main difference is that today s carrier-neutral datacenters often have more power and cooling available than the older carrier hotels or carrier points of presence (POPs). FIGURE 1: CARRIER-NEUTRAL DATACENTER COMPARED WITH MEET-ME ROOM Source: 451 Research, 2015 CHARACTERISTICS CARRIER-NEUTRAL DATACENTER MEET-ME ROOM Any size, but usually >10,000 sq. ft Almost always smaller than a carrierneutral datacenter; often 1,000-5,000 sq. Size ft Power and cooling Typically built to densities that accommodate servers and edge routers rather than less powerhungry switches Originally built for telecom equipment, they typically offer DC power and relatively low density, though many have been upgraded to handle servers and larger routers Stand-alone building Yes or No No Ownership Owned by datacenter operator, or in space leased by the operator Owned by the owner of the building Operator Datacenter owner Building management, or an operator that has a contract with the building owner Purpose Can be interconnection-focused, or focused on providing space and power with the ability to connect to multiple carriers Interconnection Policies on interconnection Size of deployment Examples Typically only allow interconnection with other tenants in the datacenter Typically a minimum deployment is required e.g., 5-10 racks with smaller amounts provided by tenants Equinix, KDDI/Telehouse, Interxion facilities Typically, any building tenant can interconnect, whether leasing space in the MMR or not Full racks, half racks, quarter racks Telx in Digital Realty facilities, 151 Front Street meet-me room operated by Allied Fiber in Toronto, CoreSite in Denver

9 5 451 RESEARCH 2.2 INTERCONNECTING THE INTERNET In the early days of computer networking, there existed many incompatible and disjointed networks (e.g., enterprise networks and government-run networks that used different proprietary networking technologies). Not only were the networks incompatible, they were created with different purposes and were not expected to interoperate. The US Department of Defense, for instance, had ARPANET, which connected different research sites, while CSNET was created for the academic and commercial community of computer scientists. Eventually, users on one network wanted access to data or wanted to exchange with users on other networks. In the early 1980s a commercial multi-protocol router was created, as were a number of exchanges where networks could interconnect and transfer traffic between different networks. These facilities were initially run by government agencies and nonprofits, and they became known as network access points, or NAPs (e.g., MAE-East in 1992). The management of these was eventually moved to commercial entities mainly telecom providers such as Sprint and some of the Regional Bell Operating Companies (RBOCs). After the original sites became too crowded, particularly as data and content moved beyond the telcos to firms such as AOL and Yahoo, other exchanges were created. This drove the growth of commercial Internet exchanges (IXs) that we see today, in the multi-tenant datacenter (MTDC) landscape. Currently, there are different methods and business arrangements for transferring data between networks at interconnection points PRIVATE INTERCONNECTION Private interconnection (or peering ) is when networks are interconnected directly between edge routers on each network. This is typically done using a pair of fiber-optic cables (one for transmitting, one for receiving), called cross-connects, and may involve running these cables from one party s equipment directly to the other s, or both parties running cables to the central meet-me room. Examples of private interconnection include transit and private peering.

10 6 Interconnection 101 Internet Transit Internet transit or IP transit refers to when an ISP sells global access to the Internet. In practice, this usually means a network, or autonomous system (AS), is paying for the ISP to announce Internet routes to it and to let the rest of the Internet know that the AS or network and its customers are on the Internet (see Figure 2). FIGURE 2: INTERNET TRANSIT Source: 451 Research, 2015 C Customer Transit $ TRAFFIC FLOW Transit $ Customer Customer A B Customer Transit traffic is Ethernet and is exchanged typically at 10Gbps or, increasingly, 100Gbps. The most common way to bill for transit is the 95/5 model. Every five minutes, the amount of traffic passing over the link is sampled. Every month, the readings are sorted from lowest to highest and the 95th percentile (of traffic either in or out, whichever is highest) is used to calculate what the customer pays, so the top 5% of spikes in traffic are not included. Thus, overall, the more transit used, the higher the costs. Transit costs vary widely but have been declining steadily for years. Current estimates range from $3/ Mbps (and up) to as low as $0.50/Mbps. Although the prices have been steadily declining, contracts tend to be for a year or so, and traffic per customer generally is rising, so the cost curve looks like the following: TRANSIT COST NO. OF MBPS

11 7 451 RESEARCH Internet Private Peering Peering is when two parties provide access to each other s network endpoints by interconnecting and exchanging routing information (see Figure 3). Peering is not used for traffic going to end users on networks other than the peers. It is referred to as private peering, because the two parties connect directly. Peering can help optimize traffic flow and latency. It is typically settlement-free, meaning that no payments exchange hands, since the two parties exchange roughly the same amounts of traffic. If there is an imbalance in traffic (e.g., one party receives more traffic than it sends), one party will pay the other for access to its customers; this is called paid peering. FIGURE 3: PRIVATE PEERING Source: 451 Research, 2015 Customer Customer Customer A Peering B Customer TRAFFIC FLOW It is not always cost-effective to peer. Setup costs for peering are typically higher than for transit, so peering is cost-effective once there is a high enough volume of traffic. There may be some setup costs for transit; for example, if the transit connection is made in a colocation facility there will be costs for renting space in the facility and possibly for network connectivity between the facility and the customer s office. For peering, there will be the same costs to be at a meeting point (often a colocation facility), plus typically the cost of a router (rather than just a switch), the setup fee for a cross-connect to the peer(s) and in some cases a monthly fee for the cross-connect(s) as well. However, once they have a cross-connect, peers can exchange as much traffic as the size of the crossconnect (well, up to 70-80% of the cross-connect size, to be safe). There are higher fixed costs, but once enough traffic is passed over the cross-connect, the cost is lower. Typical costs are anywhere from $100 to $350/month per fiber cross-connect. So if sending or receiving 500Mbps per month (95th percentile) at a transit cost of $2/Mbps, the transit cost would be $1,000 per month, while the same traffic over a cross-connect would cost $350 plus the setup costs, for a cost curve that looks more like this: TRANSIT COST CROSS-CONNECT NO. OF MBPS

12 8 Interconnection 101 However, even if it is does not necessarily save money to peer versus using transit, some firms prefer to peer in order to have traffic go directly to the peer s end users, avoiding the hops that a transit provider might send the traffic through. In other words, networks may prefer peering to gain more control over traffic routes (see Figure 4). No single ISP is physically connected to every other network on the planet; most have a customer base in a particular region. So an ISP that sells transit also has to connect with network providers via peering arrangements, IXs or by buying transit as well. Through this series of business relationships and network connections, each network can reach the entirety of other websites on the Internet, and vice versa. FIGURE 4: INTERNET TRANSIT PLUS PEERING Source: 451 Research, 2015 C Peering D Customer TRAFFIC FLOW Transit $ Transit $ Customer Customer A B Customer PUBLIC INTERCONNECTION OR PUBLIC PEERING Public peering refers to the practice of multiple parties connecting to each other via an IX that operates a shared switching fabric, typically an Ethernet switch, which enables oneto-many connections. The location and switch used to connect multiple firms is called an Internet exchange point (IXP). The Ethernet switches can provide 100Mb connections (or ports), up through 100Gb ports in some cases (see Figure 5). Public peering is more scalable and often less expensive than setting up a large number of individual private peering arrangements/connections. Once connected to the main platform, there is relatively little cost to add interconnection partners that are also on the platform.

13 9 451 RESEARCH FIGURE 5: PUBLIC PEERING PLATFORM Source: 451 Research, 2015 ISP A POP ISP B POP Router Router Internet Exchange Point Router Router PUBLIC PEERING Across a Shared Public Peering Switch Ethernet Switch Router Router PRIVATE PEERING Across a Cross-Connect Router Router ISP C POP ISP D POP In North America, in general, there is one major public peering exchange per market, typically available via one or two datacenters. The owner(s) of those datacenters typically run the exchange. The reverse is true in Europe, with most public peering fabrics operated on behalf of their members either as nonprofits or as cooperatives and available in multiple datacenters in the market. Their members are the firms connected to the exchange. This model has slightly different economics: Since the exchanges are in multiple sites, there are costs for equipment in each site and network connectivity between them (e.g., the cost to lease dark fiber and the cost for equipment to light the fiber at each end). In North America, private peering is more common; public peering has generally been used for lower bandwidth requirements and/or as a backup for private peering traffic. Public peering is more popular in Europe than in North America for historic reasons, since it arrived in Europe later, when the technology was better developed (see Figure 6).

14 10 Interconnection 101 FIGURE 6: PUBLIC PEERING IN THE US VS. EUROPE Source: 451 Research, 2015 CHARACTERISTICS US EUROPE IXP business model For-profit Cooperative or nonprofit IXP operator The colocation provider A committee selected by members or an association IXP location Interconnection price model Cross-connect price model The IXP is located in the facility (-ies) of its colocation provider. Installation fee for connection to the IXP based on number and bandwidth of ports provided, plus monthly recurring fee. Installation fee plus, often, monthly recurring fee paid to the colocation provider per crossconnect. The IXP has equipment in multiple datacenters belonging to a variety of operators. Installation fee for connection to the IXP based on number and bandwidth of ports provided, plus monthly recurring fee. There is also an annual membership fee not related to the quantity of ports or traffic. Installation fee and typically no monthly recurring fee paid to the colocation provider per crossconnect.

15 RESEARCH SECTION 3 Interconnection as a Business Originally, connections were made by physically patching (connecting) two customers together via a fiber-optic or copper cable. Every carrier in a facility was connected individually to others. Over time this generated enormous quantities of cables that were hard to keep track of and became quite complex to manage. (Physical network connections when done wrong are believed to be a major source of network errors.) In the early carrier-neutral sites, the building owner sometimes made the physical connections, i.e., ran the meet-me room. Sometimes the carriers ran the meet-me room themselves, e.g., as a cooperative. As complexity grew, firms sprang up that specialized in operating interconnection spaces. They worked out arrangements with the building owners and earned their keep by charging for their services. When the original carrier hotels filled up, these operators sometimes built and ran expansion space nearby. This launched the business of interconnection and also led to the automation of the process, when the interconnect operators began to provide switching services (as well as the physical cabling services). 3.1 COMPONENTS There are several components to the business of interconnection: The building where the connections are made In some cases, bandwidth services to or within the building where the connections are made Physical cross-connects Often, a public peering platform Access to other customers of the facility, such as cloud providers, either directly or through a cloud exchange platform Additional services provided to customers THE BUILDING In the early days of carrier-to-carrier connections, the facility where connections were made mainly housed telecom equipment which generally requires relatively little power but uses direct current (DC). Thus, when these facilities were set up in office buildings, they did not normally require extra power and cooling they just required DC plant. Through the years, as more firms sought to connect, Internet traffic grew, and customers signed on that required AC plant and more power and cooling, the facilities had to be upgraded. The owners had an incentive to do that because as the number of customers and connections grew, the facilities became more valuable.

16 12 Interconnection BANDWIDTH Customers of the facility typically need to pay for bandwidth to their offices or other sites and for transit to access Internet customers that are not on the networks of firms the customer peers with. In general, the customer sets up a direct relationship for bandwidth and/or transit with carriers in the facility. Sometimes, however, the owner of the datacenter also provides bandwidth services and can charge separately for those. In addition, some facilities are connected to others to provide access to customers in those other facilities, and bandwidth is required between the datacenters. This can be provided, on a separate contract, by a dark fiber or network service provider (NSP). Or sometimes, again, the datacenter owner/operator provides the connectivity to other datacenters either as a separate charge or rolled into one of the other fees CROSS-CONNECTS A customer pays to be in a datacenter but also needs to be connected to other firms in the datacenter. In the early days, carriers ran cables themselves but as the number of cables grew, this became unwieldy and a third party took over managing the physical cables. The third party charged a fee for this service. This fee remains in place today and typically is an installation charge to pay for a technician to physically run the cables and connect them (it also covers the cost of the cable and equipment). In addition, some providers also charge a monthly recurring fee for the cross-connect PUBLIC PEERING PLATFORM As mentioned above, an IXP allows a customer to connect to one platform and, through that platform, to other members of the exchange without having to run separate cables each time. There is a fee for this service typically an installation fee and a monthly maintenance fee as well. It is generally based on the size of the port (e.g., 1Gb per second), though some providers (e.g., IIX) charge a fee based on the amount of bits actually transferred ACCESS TO OTHER CUSTOMERS IN THE FACILITY, PARTICULARLY CLOUD PROVIDERS Some interconnect providers offer ways to connect to other customers in the facility. These may include a portal that allows customers to see and contact each other, or a cloud exchange, which in theory is a platform that allows customers to connect to multiple cloud providers easily by incorporating the APIs and specific requirements for access to each cloud provider into one platform. These are at various stages of development, depending on the provider, but can certainly be an additional source of revenue ADDITIONAL SERVICES Customers may require consulting, network management, remote hands and other services that are billed separately.

17 RESEARCH CLOUD EXCHANGE EXAMPLE: EQUINIX Cloud exchanges are still relatively new. Equinix launched its Cloud Exchange in spring of The idea is to take the IX concept and expand it beyond NSPs to connect to other infrastructure service providers. Ideally, this would allow a customer to connect to multiple IaaS providers available on the exchange such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform and SoftLayer (IBM) through one interface or portal. This has been relatively complex to set up; cloud providers have different requirements for accessing their clouds, so a portal has to provide the correct information to each provider. The Equinix Cloud Exchange does this using a co-developed version of Cisco s InterCloud orchestration tool coupled with SDN technology developed in Equinix labs, as well as components from Ciena and Juniper for layers 1-3 and the software platform Apigee. The Cloud Exchange provides a range of services, including automatic provisioning and policy setting. A customer can connect to Cloud Exchange participants via a port on an Equinix switch. Instead of taking out a dedicated fiber connection to each cloud provider, the customer can open many smaller virtual circuits to various cloud suppliers. Equinix is aiming to encourage end users to connect with providers, pricing the service as a utility to help spur connections between a customer and multiple cloud providers. Equinix, in turn, makes money from the customer and supplier for both colocation and the cross-connect to the platform, as well as a nominal fee for joining the platform. Cloud Exchange VLANs target enterprise users consuming smaller amounts of traffic for smaller time frames (200Mbps, 500Mbps, 1Gbps and other speeds up to 10Gbps are available). Those customers with higher bandwidth consumption rates over a long-term contract, including those looking at Amazon s Direct Connect service, will buy 1Gbps or 10Gbps ports. 3.2 SUPPLY AND DEMAND SUPPLY Carrier hotels today, particularly those with the most customers, in general remain more valuable than other datacenters. Typically, several carriers have laid fiber to the building while others have installed equipment inside, so there is a sunk cost to being in the facility and it can be expensive to move to another one. Customers usually need to be in the facility because they need to connect to the maximum number of carriers, ISPs, cloud providers and others. It is difficult to start a competing exchange nearby because each exchange has a tipping point before which there are not enough customers in the facility to make it worth paying for equipment and colocation fees to have a presence there. Another option is to build a datacenter nearby and connect it to the original carrier hotel. However, there is then a cost for the connectivity between the two buildings (currently around $1,000 per month for 10 Gigabit metro Ethernet connectivity, although this varies widely). The owners/managers of the original carrier hotel meet-me room have an advantage, as they can pay for dark fiber to connect the two buildings and arrange for interconnection of customers from the second facility. However, if a competitor sets up a building nearby, the competitor needs to work with the original carrier hotel owner to determine how to connect customers of the second building. Typically, the customers of the second building would need to pay for network transport to the carrier hotel. Note

18 14 Interconnection 101 that the original carrier hotel owner can simply charge a lower price for space in the carrier hotel than the cost of that network transport and try to win the customer away. Or the owner of the second building can pay for network connectivity to the first building, but then someone has to pay for space at the first building to house the equipment for interconnection. As a result, with captive customers for which moving may be expensive, and with some barriers to entry that keep competitors from easily recreating the ecosystem of customers at a facility, the carrier hotel owner/operator can have strong pricing power. The fees for colocation in the building (just for the space and power for equipment) are typically at least 20% higher than those for facilities nearby that are less network-dense. Sometimes for a very desirable location, e.g., where the matching engine for financial trades sits, the fees can be much more than that. If there is not enough capacity for growth at a particular network-dense facility, we have seen ecosystem participants move to a different location. This has happened, for example, with financial exchanges where the trading engine moved from downtown (in Manhattan or Chicago) out to the suburbs and brought its trading ecosystem participants with it DEMAND In addition to acting as hubs where network providers can connect to each other, a variety of models for interconnection have arisen between enterprises, NSPs and cloud service providers. The business model of the MTDC operator is one component to that value whether or not they can attract a large number of providers of bandwidth (ISPs, carriers and such), and customers that need connectivity to the public Internet as well as to other customers in a particular datacenter facility. A more recent factor in the equation is the presence of cloud compute service providers such as AWS, Microsoft Azure, Google Compute Engine and IBM SoftLayer as well as SaaS companies such as salesforce.com. In addition, enterprises are looking to place applications closer to an ever more mobile customer and employee base. Locating in facilities with higher peering points with mobile carriers can greatly improve application performance. This extends to enterprise partners providing services such as marketing and integration, like HubSpot, Eloqua and Marketo. Providing connectivity to these services for customers already colocated in a datacenter is an area of significant commercial activity. Another area of growing interest is secure, private connections between cloud providers and customers outside the facility. The value of an interconnection ecosystem is growing and is already very large for companies in particular sectors, e.g., where groups of companies need to share large data sets (oil and gas, movie production, pharmaceuticals and genomics), or need to trade information (financial services trading ecosystems). As more firms start to compute and share large data sets, demand for these communal meeting points (datacenters) will continue to grow.

19 RESEARCH 3.3 CUSTOMERS With the rise of the Internet, firms besides carriers have sought to connect with carriers and with each other, so the list of customers/participants at interconnection facilities has grown (see Figure 7). FIGURE 7: CUSTOMERS OF INTERCONNECTION FACILITIES Source: 451 Research, 2015 CUSTOMER DESCRIPTION REASONS FOR INTERCONNECTING Network service provider or carrier ISP Content provider Content delivery network Provides network access and very high-volume bandwidth access to the Internet backbone. NSPs sell bandwidth to ISPs, which in turn connect and sell access to consumers and enterprises; some carriers also sell directly to enterprises. Provides businesses and consumers access to the Internet. May offer other services, e.g., , website hosting. Usually a large-scale provider that stores video Web pages or other files that consumers want to access. A set of distributed servers and software used to deliver content. To make the maximum number of connections for buying and selling Internet transit, peering and VoIP interconnection. The cost savings from being in an interconnection facility usually make up for the equipment and rental costs to be there. Carrier equipment in these facilities typically requires less power than servers. The footprint is relatively fixed. To gain access to all other destinations on the Internet that they are not connected to, ISPs buy transit from network providers or peer with them or with other ISPs. They want to be in an interconnection facility with the maximum number of small networks and ISPs present in order to peer with them, as well as with top-tier network providers present in order to buy transit from them. Content providers need to interconnect with networks via Internet peering or transit to serve their content to end users. They tend to have large server and storage deployments. Often, interconnect facilities do not have enough contiguous space available for the full content deployment, so much of it ends up in a building close by, connected to the interconnect facility via dark fiber or wavelength services. Facility quality and reliability are of great importance to most content providers, as they lack carriers geographic redundancy, and, in some cases, will serve a specific product out of a single datacenter. Like content providers, CDNs will place servers at interconnection facilities in order to gain access to end users, through transit and ideally peering agreements. The direct billing model of CDNs makes them highly price-sensitive. Many CDNs are less concerned about the ability to expand within a single facility, and prefer to spread their footprint out to cover many facilities, thus improving CDN performance.

20 16 Interconnection 101 CUSTOMER DESCRIPTION REASONS FOR INTERCONNECTING Web hoster Cloud and/or hosting provider Systems integrator Enterprise A service provider that offers space on servers for websites, and enables those sites to be available to the Internet. A cloud provider is a service provider offering IaaS, SaaS or PaaS in an on-demand, multi-tenant environment. Examples include Amazon Web Services, Microsoft Azure, IBM SoftLayer and salesforce.com. A company whose business is building complete compute systems from disparate hardware and software components. Enterprise can refer to any business entity. For the purposes of this report, an enterprise is typically a company with 500 employees or more; it may have extensive WANs and its own datacenters, but buy network and compute resources from third parties in order to conduct business over (and on) the Internet. Margins for the hosting business tend to be lower than those of content providers, so hosters are often more concerned about price and less about facility quality. Web hosters are also less concerned with network density, as their content tends to be less attractive for Internet peering. They are mainly looking for lower-cost providers for Internet transit services. Similarly, network services such as dark fiber and wavelengths are of less interest to Web hosters, which typically have less traffic than many content providers. SaaS and IaaS providers tend to have larger footprints than network providers, often with relatively high-density architecture. Their growth is relatively unpredictable as well, so they often seek facilities with the capability to provide relatively large amounts of power in small footprints and space available for expansion. The major systems integrators (SIs) are interested in the use of interconnection facilities as a cost-saving measure. Customers of SIs enterprises of varying sizes will use many different network providers for connecting with their SI vendors. Placing SI infrastructure in interconnection facilities enables easy interconnection with carrier Internet, ATM and MPLS networks. Large enterprises are becoming more interested in interconnection facilities to increase their network options and access cloud and other service providers. Enterprises also sometimes use interconnection facilities as disaster-recovery hubs. In general, enterprises prefer higher-quality facilities with highly redundant cooling and power and a high level of security. They tend to be less costconscious than some other customers, as their footprints are smaller and facility quality is so important to them.

21 RESEARCH We summarize the drivers behind selection of interconnect facilities in Figure 8. FIGURE 8: DRIVERS OF FACILITY SELECTION Source: 451 Research, 2015 Criteria and level of importance CARRIERS CONTENT PROVIDERS WEB HOSTING PROVIDERS CONTENT DELIVERY NETWORKS LARGE ENTERPRISES, SYSTEM INTEGRATORS Network density High High Medium High Medium Telecom services available Medium Medium Low Medium High Cost Low Medium High High Low Power and cooling Low High High High Medium Expansion capacity Medium High High Medium Low Managed services Low Low Low Low High Facility quality, reliability Medium High Medium High High Examples Verizon, Level 3, Zayo Google, Netflix Rackspace Akamai, Limelight EDS, IBM, Morgan Stanley

22 18 Interconnection 101 SECTION 4 Interconnection Providers Originally, providers were known for particular locations where they had facilities, and in each market there were only one or two interconnection options. This has changed somewhat, as larger providers have acquired the single-site carrier hotels in various cities and/or have built competing facilities in the top markets. In smaller markets there are often no public IXs, but there are still locations where carriers, ISPs, content providers, etc., meet to exchange traffic. These exchange points are typically owned by a carrier or ISP. The Market Map in Figure 9 shows key interconnect providers and some of the characteristics that differentiate them. Geographic reach and focus is one characteristic: Some firms are in multiple countries; some are in a single region, typically a top market; some are in markets at the edge of the Internet, in cities outside the traditional top 10 datacenter/interconnection locations. When it comes to service offerings, there are providers that offer interconnection but also provide their own network services. There are firms that offer interconnection but also larger suites, in a combination of interconnection and a more wholesale-like offer. There are firms that offer connections through a public peering platform. Finally, there are firms that offer direct connectivity to cloud providers, through a cloud exchange platform or through direct connections to well-known public cloud providers such as AWS and Microsoft Azure. FIGURE 9: 451 RESEARCH INTERCONNECT MARKET MAP TM Source: 451 Research, 2015 Focus on Single Market GLOBAL NET ACCESS (GNAX) MIAMI-CONNECT MORGAN REED GROUP SIERRA DATA CENTERS MARKLEY GROUP CITYNAP COLO ATL Hosts or Operates Public Peering Platform MARKLEY GROUP CITYNAP COLO ATL DUPONT FABROS TECHNOLOGY CYRUSONE CORESITE Geographic Reach (Multiple Countries) Interconnection Plus Larger Suites MARKLEY GROUP DUPONT FABROS TECHNOLOGY CYRUSONE SABEY DATA CENTERS QTS REALTY TRUST EQUINIX KOMO PLAZA PHOENIX NAP PHOENIX NAP GLOBAL SWITCH GLOBAL SWITCH CORESITE PTT METRO PTT METRO EVOSWITCH EVOSWITCH EVOSWITCH SWITCH SUPERNAP WESTIN BUILDING EXCHANGE WESTIN BUILDING EXCHANGE GLOBAL SWITCH AMS-IX AMS-IX DIGITAL REALTY DIGITAL REALTY DE-CIX DE-CIX Focus on Markets Outside Top 10 IIX LINX IIX LINX COLT NTT COMMUNICATIONS SWITCH SUPERNAP SABEY DATA CENTERS SABEY DATA CENTERS INTERXION INTERXION PCCW EDGECONNEX 365 DATA CENTERS 365 DATA CENTERS KDDI/TELEHOUSE KDDI/TELEHOUSE EXPEDIENT DATA CENTERS KIO NETWORKS KIO NETWORKS CENTURYLINK INVOLTA COLOGIX COLOGIX LEVEL 3 COMMUNICATIONS NETRALITY PROPERTIES ZAYO/ZCOLO ZAYO/ZCOLO TELSTRA TELSTRA SUNGARD AVAILABILITY SERVICES NEXTDC NEXTDC VERIZON TERREMARK VERIZON TERREMARK TIERPOINT TATA COMMUNICATIONS TATA COMMUNICATIONS VXCHNGE Network Services & Cross-Connects SWITCH SUPERNAP KIO NETWORKS CYRUSONE EQUINIX TELX EQUINIX Direct Connections to Public Cloud Providers CENTURYLINK GLOBAL SWITCH NEXTDC LEVEL 3 COMMUNICATIONS COLT COLT KDDI/TELEHOUSE NTT COMMUNICATIONS NTT COMMUNICATIONS PCCW PCCW TELSTRA TELSTRA VERIZON TERREMARK VERIZON TERREMARK ZAYO/ZCOLO Cloud Exchange Platform ZAYO/ZCOLO INTERXION EQUINIX EQUINIX TATA COMMUNICATIONS TATA COMMUNICATIONS COLOGIX CENTURYLINK CENTURYLINK LEVEL 3 COMMUNICATIONS LEVEL 3 COMMUNICATIONS CORESITE CORESITE TELX TELX SWITCH SUPERNAP SWITCH SUPERNAP

23 RESEARCH Identification and placement of companies into these segments is based on analysis, both published and unpublished, performed by 451 Research. This analysis includes interviews, reports and advisory work with several thousand enterprises, vendors, service providers and investors annually. 451 Research Market Maps are not intended to represent a comprehensive list of every vendor operating in this market. Inclusion on 451 Research Market Maps does not imply that a given vendor will be specifically featured in one or more 451 Research reports. FIGURE 10: INTERCONNECTION PROVIDER SEGMENTS Source: 451 Research, 2015 PROVIDER FOCUS ON SINGLE MARKET FOCUS ON MARKETS OUTSIDE TOP 10 GEOGRAPHIC REACH (MULTIPLE COUNTRIES) INTERCONNECTION PLUS LARGER SUITES NETWORK SERVICES AND CROSS- CONNECTS HOSTS OR OPERATES PUBLIC PEERING PLATFORM 365 Data Centers ü ü AMS-IX ü ü CLOUD EXCHANGE PLATFORM DIRECT CONNECTIONS TO PUBLIC CLOUD PROVIDERS CenturyLink ü ü ü ü CityNAP ü ü Colo Atl ü ü Cologix ü ü ü Colt ü ü ü CoreSite ü ü ü ü CyrusOne ü ü ü DE-CIX ü ü Digital Realty ü ü DuPont Fabros ü ü EdgeConneX ü Equinix ü ü ü ü ü Evoswitch ü ü ü Expedient Data Centers Global Net Access (GNAX) ü ü Global Switch ü ü ü ü IIX ü ü Interxion ü ü ü Involta ü KDDI/Telehouse ü ü ü KIO Networks ü ü ü KOMO Plaza Level 3 Communications ü ü ü ü ü LINX ü ü Markley Group ü ü ü

24 20 Interconnection 101 PROVIDER Miami-Connect Morgan Reed Group Netrality Properties FOCUS ON SINGLE MARKET ü ü FOCUS ON MARKETS OUTSIDE TOP 10 ü GEOGRAPHIC REACH (MULTIPLE COUNTRIES) INTERCONNECTION PLUS LARGER SUITES NETWORK SERVICES AND CROSS- CONNECTS HOSTS OR OPERATES PUBLIC PEERING PLATFORM CLOUD EXCHANGE PLATFORM DIRECT CONNECTIONS TO PUBLIC CLOUD PROVIDERS NextDC ü ü ü NTT Communications ü ü ü PCCW ü ü ü Phoenix NAP ü ü PTT Metro ü ü QTS Realty Trust Sabey Data Centers Sierra Data Centers SunGard AS ü ü ü ü ü ü Switch SUPERNAP ü ü ü ü ü Tata Communications ü ü ü ü Telstra ü ü ü ü Telx ü ü ü TierPoint ü Verizon Terremark ü ü ü ü vxchnge Westin Building Exchange ü ü Zayo/zColo ü ü ü ü ü