C H A N G I N G B U S I N E S S P R I O R I T I E S I N A S H I F T I N G W O R L D

Transcription

1 W H I T E P A P E R T h e I n t e l l i g e n t W A N : I T T r a n s f o r m a t i o n, C o n t e n t, a n d B i g D a t a P o s e B i g T r a f f i c C h a l l e n g e s Sponsored by: Level 3 Communications and Alcatel-Lucent Lucinda Borovick February 2012 Richard L. Villars Global Headquarters: 5 Speen Street Framingham, MA USA P F E X E C U T I V E S U M M A R Y Over time, wide area network (WAN) consumption patterns have evolved from predictable and point-to-point transaction-oriented activities to bandwidth-hungry, always-on, many-to-many interactions. Virtualization, content such as videos and unstructured files, and Big Data analytics are the newest entrants to the corporate WAN and are causing a Big Traffic issue. For many organizations, the WAN is prioritized to the point of being "untouchable." While this strategy has contributed to the stability of the network, it has created a complex web of network equipment and services that will be insufficient to handle the next generation of wide area network workloads. Unfortunately, very few organizations have taken a holistic, systemsbased approach to the wide area network. IDC believes that as the content and data become "bigger," leading organizations must take a proactive approach to the wide area network. Today's WAN must include the following elements: Active/active architectures to support uninterrupted operations High performance to facilitate the distribution of large file sets and a massive number of small file sets in real time Network intelligence to keep up with business demands for capacity and qualityof-service expectations A coordinated approach to planning implementations across multiple datacenters and remote locations with these elements in mind will ensure that the WAN enables businesses to capitalize on the explosion of content and analytics to come. C H A N G I N G B U S I N E S S P R I O R I T I E S I N A S H I F T I N G W O R L D Companies rely on an expanding set of applications to compete in today's rapidly evolving business environment. For example: They rely on a fast-growing array of applications and devices ( , collaboration tools, and smartphones/tablets) to communicate and conduct business with customers and business partners.

2 They are digitizing records, design documents, and other types of unstructured data to boost efficiency, offer new services, and comply with evolving government regulations. They are collecting, storing, and analyzing more information (machine-generated data) about products, customers, and transactions. This expansion in the range of applications and information that companies are creating, accessing, and reusing has a significant effect on how they organize their datacenters and move information between those datacenters. CIOs must place a laserlike investment focus on IT solutions that address three business requirements: Improve business agility and reduce the cost of doing business through aggressive use of technologies such as virtualization in datacenters Ensure the integrity of the business and its information assets in the face of natural disaster, datacenter disruptions, or local system failures through greater use of cloud solutions Enable more effective use of all types of information for a wider range of users by making it easily, but securely, accessible and minable using Big Data solutions T r a n s i t i o n t o t h e W o r l d o f V i r t u a l i z e d I T 2011 was a pivotal year in the use of server virtualization technology within datacenters around the world. IDC believes that 2011 was the first year in which more application instances (new and old) were deployed as virtual machines on a virtualized server rather than on a dedicated physical server. The use of virtualization for application servers will only accelerate after 2011, surpassing 75% of deployed applications within a few years. A primary requirement for further expansion in the use of virtualization is to improve the mobility of virtualized applications within a single datacenter and, more importantly, across dispersed datacenters. IT teams want to nondisruptively move virtualized applications and their supporting data sets between datacenters to address three goals: Disaster recovery: fast recovery of applications in case of an unanticipated datacenter failure Disaster avoidance: premigration of applications to a distant datacenter in anticipation of a potential failure (e.g., approaching hurricane) Datacenter optimization: regular transfer of applications to new datacenters to reduce costs (e.g., follow the moon to reduce energy costs) or enhance performance (e.g., follow the sun to minimize impact on latency-sensitive transaction applications) To meet these goals, IT teams must plan for a major expansion of data replication WAN links (traditionally limited to mainframe and mission-critical transaction applications) between datacenters in terms of both capacity requirements and diversity of traffic flows. 2 # IDC

3 R i s e o f t h e C o n t e n t - D r i v e n E n t e r p r i s e While aggressive use of virtualization is reducing the rate of growth in server deployments in datacenters, the creation, organization, and distribution of files and rich content are driving a rapid and sustained increase in storage deployments. In 2011, organizations around the world deployed 15,900 petabytes of new disk storage capacity for the storage of file-based data. By 2015, new storage capacity deployed annually in datacenters to store all this content will reach 86,300 petabytes (see Figure 1). F I G U R E 1 W o r l d w i d e E n t e r p r i s e D i s k S t o r a g e C o n s u m p t i o n 120, ,000 80,000 (PB) 60,000 40,000 20, Total f ile-based PB Total block-based PB Source: IDC, 2012 This increasing dependence on unstructured data types (e.g., documents, records, images, and video) for competitive differentiation and more effective operations is posing a number of challenges for IT and storage administrators: Inefficient use of storage assets within and across datacenters due to uneven capacity and performance demands (low utilization overall with a few performance hotspots that fluctuate regularly) Inefficient, unreliable, and increasingly impractical backup and disaster recovery practices that are incapable of handling recovery of very large (multi-petabytes) data sets For many small and midsize businesses, this content explosion will lead to greater use of public cloud based storage services for tasks such as backup, archiving, and data sharing. Larger enterprises, with more pressing concerns about data ownership and privacy, will need to deploy new classes of information solutions that are optimized for rapid data ingest, extremely large storage capacities, and consistent availability within single datacenters and across dispersed datacenters around the world IDC #

4 The need to transport very large content-oriented data sets between datacenters is a new, but increasingly critical WAN networking challenge for organizations in sectors such as media/entertainment, healthcare/life sciences, and security (video surveillance). Making things more challenging is that these transfers often cross corporate boundaries (e.g., a media company distributing content to movie theaters or public cloud services such as Netflix or itunes). These new conditions increase the need for highly flexible, yet secure WAN infrastructure. B i g D a t a : I n f o r m a t i o n E v e r y w h e r e, B u t W h e r e ' s t h e K n o w l e d g e? For the first 40 years of the IT industry, the main data challenge for most organizations was enabling/recording more and faster business transactions, often referred to as structured data. Today, much of the focus is on more and faster exchanges of information (e.g., documents, medical images, movies) between cloud clusters and PCs, mobile devices, or living rooms. Semistructured data is often overlooked, but with the advent of RFID tracking, smart sensors, mobile devices with geospatial information, and a growing array of data collection devices, machine-generated data will be a leading driver of the data explosion. IDC believes that organizations that are best able to make real-time business decisions based on machine-generated data streams at the lowest possible cost will thrive, while those that are unable to embrace and make use of this expanding data source will increasingly find themselves at a competitive disadvantage in the market. Making investments to quickly extract value from this expanding pool of information is fast becoming a core business mandate (often tied to the term "Big Data"). Such efforts can quickly lead to spiraling IT and network costs along with growing corporate risk associated with data proliferation and movement. In fact, development of Big Data solutions at enterprise scale will not be successful unless IT organizations provide the WAN architecture to reliably move data to and from private and public datacenters, where this data resides and is continually analyzed. V I R T U A L I Z A T I O N, C O N T E N T, A N D B I G D A T A D R I V E B I G W A N T R A F F I C C H A L L E N G E S The three major IT developments (virtualization, content/cloud, and Big Data) each pose significant challenges from the perspective of WAN performance, reliability, and flexibility. The real problem for most IT organizations is that while all are occurring concurrently, strategies for addressing each occur in isolation. IDC is already hearing from a number of IT leaders that such stovepiped approaches are beginning to drive up costs, threaten reliability, and reduce IT responsiveness to business demands. The remainder of this white paper looks more deeply into how all of these IT developments are altering wide area network requirements and discusses what organizations can do to overcome or avoid these potential roadblocks. 4 # IDC

5 W A N a t t h e B r e a k i n g P o i n t IDC finds that organizations that have upgraded their datacenter network and wide area network to support virtualization and cloud initiatives are on the road to supporting Big Data as well. Nonetheless, the combination of needs for these three trends requires today's network manager to adopt a holistic approach to handle this "Big Traffic" because each trend has slightly unique requirements. For instance: Big Data requires essential changes in the compute and storage infrastructure to provide the necessary resources to make these decisions in real time. Over and above bandwidth, real-time analytics requires an organization to have the ability to process massive amounts of diverse data and have it move quickly and efficiently throughout the network. As a result, networks that are not built for both consistent low latency and line rate performance are a hindrance to successful projects. Big Data also requires fundamental changes in both the wide area network and the datacenter network to support the required levels of throughput, intelligence, and agility. The increasingly mission-critical nature of unstructured data types (e.g., documents, images, and video) requires a WAN that provides backup and disaster recovery in accordance with the new "tier 1" status of this network workload. As important, if not more so, is the network intelligence to accurately prioritize content to ensure that applications are aligned with the business. As the cornerstone of most IT strategies today, virtualization and cloud require a fundamental change to the network. Network intelligence needs to evolve to support the virtual machine and provide nondisruptive migration of virtualized applications between datacenters. The agility and cost savings promise of cloud computing is diminished if this workload migration requires static bandwidth connections and the need to overprovision bandwidth based on peak load requirements. Individually and in combination, these workloads are bringing the WAN to the breaking point. Content and Interactive Network of Things Most enterprise WAN connections are via a T1. For many organizations, "the pipe" is full and/or expensive. New video and Big Data workloads are creating challenges for the WAN. Video is no longer a "lunchtime" headache for the network manager. While 590 million consumers accessed video from work in 2011, business use cases for video are growing. Executive communications, corporate training, and marketing launches are activities that are now incorporating video. For 2012, IDC expects the situation to be exacerbated by the use of video at the desktop and on mobile devices by more users especially with smartphones and media tablets. IDC sees the media tablet, in particular, as the device with the most promise for provisioning a true, unified user interface for unified communications and collaboration purposes. The growing number of desktop and mobile video users will present increasing scalability challenges for networks, which already are stressed to support the existing (and growing) number of telepresence and room-based video systems IDC #

6 An emerging challenge to moving Big Data projects from the pilot stage to driving business value is found at the edges of the WAN. Many organizations barely have visibility to the devices sitting out in remote locations, let alone the ability to collect information from those devices. With billions of devices and over 60 billion sensors on the network delivering machine-generated data streams, adding a completely new workload to the WAN in which data is collected from devices on the network and subsequent business analytics are delivered back to the end user will bring the WAN to the breaking point. The WAN is challenged to handle large files and machine-generated data streams. Additionally, most organizations report varying levels of bandwidth available across the organization. Some sites, which may be located in major metropolitan areas, have 10 Gigabit Ethernet, while sites located in other geographies may be relegated to dial-up speeds. Further, video quality can be linked to brand image, and poor video performance reflects badly on the corporate brand. Lack of Context in Network Intelligence The network is sophisticated; highly developed routing protocols are in use every second or every millisecond to maintain network uptime and understand traffic volumes. The problem is that, in all but the most advanced environments, the network doesn't usually report actionable information that is highly correlated to the business application back to the enterprise network manager. As a result, the ability to leverage what the network "sees" is minimal to nonexistent. The network needs to deliver actionable information over and above traffic levels. It needs to distinguish between and prioritize the enterprise workloads that the business values highly and deliver the combination of network attributes to fit the workload; for example, low latency for data replication and high bandwidth for images. This lack of context carries over to the network's inability to adjust on demand to changes in bandwidth needs. Organizations are caught between two unacceptable choices. One option is to overprovision WAN bandwidth to protect the network from known or unexpected spikes in network traffic. Unfortunately, with this option, the organization is paying for overprovisioning to keep network bandwidth available but idle. The other option is to provision and pay for network services based on the average workload and risk network performance or, worse, network failure. Rigid WAN Architectures Currently, the WAN is rigid and unable to handle changing traffic patterns. Although the WAN is a complex interconnection of devices and protocols, network managers could previously count on traffic patterns being fairly predictable. For example: The vast majority of traffic emanating from the remote branch was directly routed to headquarters/datacenter locations. Similarly, datacenter backup and recovery architectures were designed with direct communications between two (or three) datacenters. The challenge for network managers is that the traffic patterns are becoming more and more complex. This is an architecture that IDC describes as "many to many" (see Figure 2). Application and user-generated traffic needs to flow in many directions 6 # IDC

7 in order to support mobile workers, cloud services deployments, and the rapidly increasing number of machines on the network that drive Big Data applications. Further, the current network architecture is not designed to handle these new patterns, and it is not able to scale to accommodate future traffic changes. F I G U R E 2 N e x t - G e n e r a t i o n M a n y - to- M a n y N e t w o r k s Private Cloud/ Datacenter HQ Regional Office Branch Office, Mobile Worker Internet Cloud Services Management Provisioning Security Cloud Applications CRM ERP Source: IDC, 2012 Accidental Architectures A holistic, systems-based approach to the network is lacking. As end-user mobility increases, content types explode, and more devices hit the network, the WAN itself is built up "ad hoc." Typical scenarios play out like this: A customer expands into a new geography and adds a link. The organization reaches a bandwidth limit and buys more bandwidth. A new compliance strategy is implemented, and the company adds bandwidth and connections to new datacenters. A new business unit is formed, and the company duplicates the network. IDC has witnessed this piecemeal approach to the WAN by many organizations. In the era of Big Data, this approach will lead to a network breakdown. It will be practically or financially impossible to manually address the bandwidth and connectivity demands. Staffing Shortages Enterprise applications and IT services are the lifeblood of most enterprises because they govern key business processes and enable critical interaction with customers, suppliers, and partners. But as the number and the complexity of applications increase, the demand to deliver IT services to the organization continues to grow IDC #

8 Unfortunately, IT and network budgets are unable to grow at the same rate as demands for new IT services a fact particularly true given today's global economic challenges and as a result, IT staffs are being asked to do more with less. A recent IDC study of datacenter managers showed that more than three-quarters (76.8%) of staff time is spent maintaining the environment and less than a quarter (23.2%) is spent on value-added activities. This challenge carries over to Big Data. One of the challenges to wide-scale Big Data projects is the significant shortage in data specialists often referred to as data scientists. This shortage can potentially be a barrier to the accelerated use of data analysts, who can advance the capabilities and proliferation of Big Data technologies and techniques. The job of a data scientist is to bring together business requirements and data analysis. Understanding business needs and leveraging data from a variety of proprietary and public sources is important. Even more critical is the ability to do so economically, so as to derive long-term strategic organizational value. Because there is a shortage of data scientists, there is a knowledge gap on how best to use Big Data to architect an efficient WAN. The possibilities are wide open to analyze network data and use this analysis to architect the most efficient WAN. Organizations that can analyze WAN traffic patterns or work with a partner that can are likely to derive competitive advantage, leading to new sources of revenue, cost avoidance, and improved profitability. IDC believes that the challenges will escalate for network managers as virtualization, content, and Big Data projects gain in prominence and mission criticality. B E N E F I T S T O A R C H I T E C T I N G A N E T W O R K I N S U P P O R T O F V I R T U A L I Z A T I O N, C O N T E N T, A N D B I G D A T A It is clear that the mix of workloads on the enterprise network will continue to expand and be driven by virtualization, large content files, and Big Data initiatives. These workloads will become the foundation of the business and will generate Big Traffic. With planning, the network will further support business goals and has the potential to help CTOs respond to the emerging challenge of transforming IT from a necessary operational expense to one that helps increase shareholder value. N e t w o r k T h a t S c a l e s S e a m l e s s l y The rapid pace of change in the types of devices, breadth of applications, and number of network endpoints will only accelerate. This pace requires a WAN that will scale seamlessly. Initial virtualization use cases centered on server consolidation for underutilized applications; but as virtualization usage expands, the WAN must adapt to this new workload. With Big Data projects, most organizations are starting with pilot projects and taking what they learn to fuel larger, more mission-critical efforts. As such, by taking a holistic, system-based approach to the network, they will be in the enviable position of scaling from the very small to the very large. It is imperative that they choose the appropriate network from the start. Initially investing in a small WAN fabric will enable the network to fluidly grow with CPU and storage growth. This modular approach brings simplification to customers and reduces operational costs. 8 # IDC

9 Part and parcel with incremental scale is dynamic resource provisioning across the WAN. Separately and in combination, virtualization, content, and Big Data will have a direct impact on enterprise network intelligence. The ability to automatically reconfigure the network in reaction to shifting network loads and failed links (requiring zero administration for additional switching or routing infrastructure) makes the network more agile. The result is a significantly reduced administration burden on the operations team, which minimizes the risk of errors and improves resiliency. A c t i v e / A c t i v e D e p l o y m e n t s T h a t B r i n g R e s i l i e n c y t o B u s i n e s s As Big Data moves from primarily supporting pilot projects to real-time data analysis that drives company revenue, the need to implement active/active network topologies is heightened. Likewise, business video and files are on the front line of online properties, corporate image, and compliance practices. This traffic on the network is not "nice to have"; rather, it is business critical. A large global financial services firm that IDC recently spoke with reported that "it is no longer acceptable to provide active/active connectivity for just our tier 1 applications. We have already consolidated and rationalized our portfolio. Everything we do needs active/active resiliency." Leading organizations are anticipating node failures. In the event of a failure, traffic is switched to another operable node for every workload. Multidimension Network Intelligence The addition of virtualization, content, and Big Data workloads requires a WAN that not only can handle high-volume traffic but also has the intelligence to recognize and report on the multidimensional aspects of the workload. Multidimension network intelligence includes bandwidth requirements, business priority, latency sensitivity, and time sensitivity. Balan ce Traffic Lo ad Bet ween Datacenters Network optimization for revenue-generating workloads has never been more important. Load balancing, caching, and additional network intelligence functionality are the keys to ensuring the network facilitates revenue-generating video and content. Content and analytics need to be treated as any other mission-critical traditional transactional system. These workloads are now on the front line of the business. As a result, the traffic must be load balanced between datacenters so that the most appropriate content, from the most appropriate datacenter, is delivered. R o u t e D i v e r s i t y Route diversity is the first step in preparing the network to be resilient in the face of unforeseen natural disasters, everyday accidents, or human error. It is important for companies to work with their service provider to understand the location and routes of its installed network equipment. The provider needs to ensure that every packet will reach its ultimate destination, even if a router or switch fails or a fiber line is cut IDC #

10 As video, analytics, and machine-to-machine applications proliferate, it is critical that network architectures and underlying software policy management be available to accommodate what will be an explosion in the number of transactions occurring in the network. Resiliency of the WAN is accomplished by working with a service provider that has planned for this. H i g h P e r f o r m a n c e First and foremost, it is important to work with a service provider that is deploying 40G and 100G products in its IP networking and optical portfolio. The complexity of highvelocity capture, discovery, and analysis requires that architects look beyond pure bandwidth. To ensure that the network delivers on high performance, network architects test network throughput between datacenters and other endpoints. Performance is necessary for data ingestion, whether it is in bulk, microbatching, or streaming. Additionally, many Big Data projects try to pull in data from transactional systems. These are real-time revenue-generating systems, and Big Data analysis has only a limited window, typically in the overnight hours, to pull from these older systems. Further, the system may reside in another company-owned datacenter or a service provider's datacenter. When a provider designs the WAN to handle the process of extracting, transforming, and reloading data quickly, the network becomes an enabler to furthering Big Data projects. Network throughput is paramount to the successful implementation of Big Data in transactional environments. C A S E S T U D Y : L E A D I N G B A N K A N D F I N A N C I A L S E R V I C E S C O M P A N Y IDC recently spoke with a retail banking organization that offers a full spectrum of banking, trust, and investment services. The bank employs over 3,000 people and prides itself on being able to tailor its financial services to the unique requirements of its customers. B a c k g r o u n d The bank is growing rapidly and expanding outside of its initial southwest region to now have a significant presence on both U.S. coasts. The bank's leaders recognize that they need to gain a better understanding of customer transactions and interactions in order to develop new products, enhance the customer experience, and mitigate risk. IT is delivering the necessary WAN and IT infrastructure to a multiyear initiative to bring additional business intelligence and reporting. These business intelligence systems are collecting and correlating data from all aspects of the business, including deposits, loans, and brokerage. 10 # IDC

11 C h a l l e n g e a n d S o l u t i o n The senior vice president of IT operations reported that multiple drivers are causing the company to significantly increase its WAN and LAN capacity. Drivers include the following: Addition of a new state-of-the-art tier 1 datacenter and the resulting need to provide real-time data transmission between multiple datacenters Internet usage that is "skyrocketing" ("In the last three to four years, the Internet usage has gone up tenfold within the company.") The challenge of delivering content to roughly 100 end sites on the network, including remote branch locations Supporting the high-bandwidth requirements of newly introduced desktop video and videoconferencing of streaming content (The bank values video not just for travel cost savings but as a productivity tool for the organization.) Deploying voice over IP systems, where the traffic is time sensitive and requires the appropriate quality of service Adding new Big Data projects to the list of mission-critical systems that need replication and disaster recovery Backup of 100 sites over the WAN Timely and efficient security and application updates for over 4,000 desktops The need for data analytics and real-time data has "gone through the roof" in support of business intelligence systems The most recent stress to the enterprise WAN was the disaster recovery and replication requirements for the new projects in combination with the mission-critical production systems. The senior vice president of IT operations reported that the "data needs were really off the charts" from anything the bank had seen previously. The bank was unable to synchronize the data on the existing DS3 or OC-3 WAN links. Additionally, the bank was challenged because the existing WAN infrastructure provided fixed bandwidth, and any attempts to increase capacity took significant effort and very often required months of planning and construction projects. The bank decided to introduce Ethernet WAN circuits. It was able to upgrade from 155 megabits per second on the OC-3 circuits to 2 gigabits worth of Ethernet capacity. The senior vice president of IT operations claimed that because he kept the deployment simple, he had the flexibility he needed to the job done. "It wasn't anything fancy. We didn't use any exceptional technologies; we just increased the amount of bandwidth tenfold between datacenters." 2012 IDC #

12 R e s u l t s The senior vice president of IT operations reported that he values the flexibility and scale that Ethernet brings to the organization. He reported that "to some degree, it's really just a phone call that we can make to increase the amount of bandwidth that we can get from our carriers." One of the bank's primary carriers is Level 3 due to its ability to provide customized solutions to the bank. C H A L L E N G E S / O P P O R T U N I T I E S Given the expected increase in virtualization, content, and Big Data, it is critical that IT organizations prepare their WAN for Big Traffic. The ability to both fit into existing network topologies and ensure that the network provides a seamless path to expand is paramount. IDC believes that organizations need to be able to treat diverse traffic types on the network. Workloads such as video may initially enter the network as a low priority and expand to require mission-critical status. Likewise, Big Data projects may be small in scope today but will likely expand over time and require scalability as well as tier 1 network attributes. The opportunity for network managers is to take a holistic approach that will have downstream benefits in the likes of resiliency and ease of management as well as cost-effective, pay-as-you grow economies. C O N C L U S I O N Virtualization, content, and Big Data are revolutionizing the way global organizations communicate, generate revenue, and adhere to compliance policies. These trends represent a major evolutionary step related to the broad objectives of furthering the intelligent economy and offering new opportunities for proactive organizational decision making. As pilot and implementation plans are considered, IDC believes that enterprises must plan for the WAN support of terabyte-sized structured and unstructured data sets. IT organizations will need to consider a coordinated approach to planning implementations across multiple datacenters and remote locations. Most importantly, they need to develop a WAN strategy that provides incremental scale as workloads grow and delivers the resiliency necessary to support mission-critical processes. IDC believes that well-developed plans for network support of Big Traffic need to address connections within the corporate network as well as anticipate the future requirement of connectivity into broader, publicly available or available-for-purchase data sets. Accomplishing this will require a number of considerations, including the need for high performance, active/active network intelligence, and route diversity. Addressing this variety of network considerations will be fundamentally necessary if customers plan to build a solid foundation for Big Data applications within their organization and will be a requirement for meeting the concerns imposed by data governance and regulatory requirements. 12 # IDC

13 C o p y r i g h t N o t i c e External Publication of IDC Information and Data Any IDC information that is to be used in advertising, press releases, or promotional materials requires prior written approval from the appropriate IDC Vice President or Country Manager. A draft of the proposed document should accompany any such request. IDC reserves the right to deny approval of external usage for any reason. Copyright 2012 IDC. Reproduction without written permission is completely forbidden IDC #