1 Networking in the Big Data Era Nelson L. S. da Fonseca Institute of Computing, State University of Campinas, Brazil IFIP/IEEE NOMS, Krakow May 7th, 2014
2 Outline What is Big Data? What is the role of networking in Big Data? What are the sources of Big Data? What are the issues in networking for Big Data? How can Big Data be processed, transferred and storaged in a friendly way?
3 What is Big Data?
6 In 60 seconds.. https://plus.google.com/+avinash/posts/mgyatu6mbhd
9 It is not just about Volume!
10 Big Data Siewert, S. B. Biga data in the cloud, IBM Developerworks, Tech. Rep., July 9, 2013.
11 Big Data and Enterprise Analytics: The real-world use of big data: How innovative enterprises extract value from uncertain data, Executive REport, IBM Institute for Business Value
14 Infrastress Alibaba Mall processes in a single day (Nov 11th, 2013) million online transactions from 213 million users and 4.1 billion transactions
16 Networking Computing Network Bandwidth Communication delays (tolerance) Degree of interactivity Storage
17 Networking Computing Network Bandwidth Communication delays (tolerance) Degree of interactivity Storage
18 What are the sources of data?
20 Map-Reduce Facebook Trace analysis: 30% to 50% of running time took up by communication phase
22 Schedulers which are data-location aware to decrease network traffic as well as I/O operation How to schedule tasks with heterogeneous (CPU, I/O) demand to promote load balance? How to benefit from Yarn resource management?
23 Scientific Computation The Montage application created by NASA/IPAC stitches together multiple input images to create custom mosaics of the sky.
24 Current cloud tools do not provide an out-of-box solution to address application needs Interconnects is the major obstacle to cloud computing broad adoption for larger-scale, more tightly coupled HPC applications
26 Sensing-as-a-Service C. Perera, A. Zaslavsky, P. Christen and D. Georgakopoulos, "Sensing as a service model for smart cities supported by Internet of Things", TRANSACTIONS ON EMERGING TELECOMMUNICATIONS TECHNOLOGIES, 2014; 25:81 93
27 Sensing-as-a-Service C. Perera, A. Zaslavsky, P. Christen and D. Georgakopoulos, "Sensing as a service model for smart cities supported by Internet of Things", TRANSACTIONS ON EMERGING TELECOMMUNICATIONS TECHNOLOGIES, 2014; 25:81 93
28 Objects may require to be uniquely identified, or to be identified as belonging to a given class Multi-service platform Distributed processing of data traces Distributed flow control Privacy
29 What are the issues in networking for Big Data?
30 Data Centers
31 Cloud Data Center Traffic Cisco Global Cloud Index: Forecast and Methodology,
32 Canonical Data Center Architecture Core (L3) Aggregation (L2) Edge (L2) Top-of-Rack Application servers
33 Data Center Traffic Most of the flows are small in size (< 10 KB) Most of the bytes in top 10% large flows Traffic leaving edge switches ON-OFF, lognormal distributions Packet size distribution bimodal (200 to 1400 B) T. Benson, A. Akella, and D. A. Maltz Network traffic characteristics of data centers in the wild. In Proc of the 10th ACM SIGCOMM conference on Internet measurement (IMC '10). ACM, New York, NY, USA,
34 Data Center Traffic In cloud data center majority of flows stay in rack (80%) while in enterprise and university data center it varies from 40% to 90% Core layer most utilized, edge layer lightly utilized Core layer contain hot spot but less than 25% of links No need for more bisection bandwidth Most of losses occur in links with low utilization due to bursty traffic
35 VM Processes VM arrival and departure processes self similar, power law VM in the system: ARIMA model Yi Han, Jeffrey Chan and Christopher Leckie. Analysing Virtual Machine Usage in Cloud Computing. In Proceedings of the IEEE rd International Workshop on Performance Aspects of Cloud and Service Virtualization, 2013
36 Need of longitudinal study on cloud (data center) traffic characterization Need of publicly available traces
37 Data Center Network Fat Tree Dcell B-Cube Jellyfish
38 Liberate upper layer switches for load balance to avoid few hot switches being overloaded Networks should guarantee well isolation, and stable service among multiple tenants.
39 Hybrid Data Center Networks Christoforos Kachris and Ioannis Tomkos "Optical interconnection networks for data centers", ONDM 2013
40 Hybrid Data Center Networks Christoforos Kachris and Ioannis Tomkos "Optical interconnection networks for data centers", ONDM 2013
41 Need of high radix, scalable, energy efficient Data Centers that can sustain the exponential increase of the network traffic.
42 VM Placing Non- trivial network topology for scalability and reliability Multi-path routing; route can change dynamically Heterogeneous services; large variety of run-time traffic pattern Unpredictable traffic variability due unpredictable request spikes and servisse-dependente operations
43 Need for traffic-aware VM placing that takes into consideration the correlations of VM traffic as well as traffic variability; dynamic placement decision.
44 VM Migration Improvement of data and network locality; Not Always possible to mantain same IP address, leading to service disruption; WAN Migration: trade off bandwidth x downtime R. Boutaba, Q. Zhang and M. F. Zhani. Virtual Machine Migration in Cloud Computing Environments: Benefits, Challenges, and Approaches. In Communication Infrastructures for Cloud Computing. H. Mouftah and B. Kantarci (Editors). IGI-Global, USA. pp , September, 2013
45 Transport protocols to handle service disruption Sophisticated management strategies for large scale VM deployment Development of Inter-data centers VM migration framework VM migration to facilitate the collaboration between cloud and mobile devices
46 How can Big Data be processed, transferred and storaged in a friendly way?
47 Software Defined Data Center
48 Software Defined Data Center <http://youtu.be/uwb4kmghzaa>
49 Virtual Networks VN Scheme Description Encapsulation Scalability- # of VNs VLAN VXLAN NVGRE Bridges VMs, for dedicated management MAC-in-IP 2 12 Ammeliorates scalability for cloud environments Ammeliorates scalability for cloud environments MAC-in-UDP MAC-in-GRE Contrail Uses Openflow All 2 12 NSX Uses Openflow All 2 12
50 Network Virtualization Routing Protocol Multicast Tree Encapsulation TRILL IS-IS Single MAC-in-MAC SPB IS-IS Multiple MAC-in-MAC NetLord SPAIN Single MAC-in-(IP+MAC) Openflow All Single All
51 OpenFlow Switching OpenFlow Switch specification OpenFlow Switch Controller PC sw hw Secure Channel Flow Table The Stanford Clean Slate Program
52 ElasticTree [Brandon Heller, NSDI 2010]
53 SDN functional architecture
54 Open Daylight Plataform
55 SDN and Hadoop P. Qin, B. Dai, B. Huang and G. Xu, Bandwidth-Aware Scheduling with SDN in Hadoop: A New Trend for Big Data, in Proc. of INFOCOM 2014
56 Multipath solution for fast traffic rerouting Scalability - current support of 10+6 flows for data centers Delay in flow set up, proliferation of flow table can limit scalability How to place a given number of controller in a certain physical network such that predefined objectives are achieved?
57 Network Programming languages Language FML Frenectic Nettle Netcore Procera Pyretic Flog HFT FlatTire Short description high level policy description language (e.g. access control) avoid race conditions through well defined high level programming abstractions allow programmers to deal with streams instead of events means for expressing packet-forwarding policies in a high level high level abstractions to describe reactive and temporal behaviors specify network policies at a high level of abstraction, offering transparent composition and topology mapping combine ideas of FML and Frenetic, providing an event-driven and forward-chaining logic programming language enables hierarchical policies description with conflict-resolution operators, well suited forn decentralized decision makers enables hierarchical policies description with conflict-resolution operators, well suited for decentralized decision makers
58 Support of SDN language to the automation of Software Defined Data Center Abstraction of resources for the support of application requirements
59 Software Defined Storage
60 Final Remarks Need of characterization of traffic generated by Big Data applications Use of communication patterns of Big Data processing to define resource allocation Distributed processing of Big Data Integration of SDN functionality into automationof SDDC for the support of requirements of Big Data applications
Network Virtualization for Large-Scale Data Centers Tatsuhiro Ando Osamu Shimokuni Katsuhito Asano The growing use of cloud technology by large enterprises to support their business continuity planning
SDN and Data Center Networks 10/9/2013 1 The Rise of SDN The Current Internet and Ethernet Network Technology is based on Autonomous Principle to form a Robust and Fault Tolerant Global Network (Distributed)
Testing Network Virtualization For Data Center and Cloud VERYX TECHNOLOGIES Table of Contents Introduction... 1 Network Virtualization Overview... 1 Network Virtualization Key Requirements to be validated...
Virtual Machine in Data Center Switches Huawei Virtual System Contents 1 Introduction... 3 2 VS: From the Aspect of Virtualization Technology... 3 3 VS: From the Aspect of Market Driving... 4 4 VS: From
24.05.13 TRILL for Data Center Networks www.huawei.com enterprise.huawei.com Davis Wu Deputy Director of Switzerland Enterprise Group E-mail: email@example.com Tel: 0041-798658759 Agenda 1 TRILL Overview
Avaya VENA Fabric Connect Executive Summary The Avaya VENA Fabric Connect solution is based on the IEEE 802.1aq Shortest Path Bridging (SPB) protocol in conjunction with Avaya extensions that add Layer
IMPROVING LINK UTILIZATION IN DATA CENTER NETWORK USING NEAR OPTIMAL TRAFFIC ENGINEERING TECHNIQUES L. Priyadharshini 1, S. Rajanarayanan, M.E (Ph.D) 2 1 Final Year M.E-CSE, 2 Assistant Professor 1&2 Selvam
Applying SDN to Network Management Problems Nick Feamster University of Maryland 1 Addressing the Challenges of Network Management Challenge Approach System Frequent Changes Event-Based Network Control
Optical interconnection networks for data centers The 17th International Conference on Optical Network Design and Modeling Brest, France, April 2013 Christoforos Kachris and Ioannis Tomkos Athens Information
Lecture 7: Data Center Networks" CSE 222A: Computer Communication Networks Alex C. Snoeren Thanks: Nick Feamster Lecture 7 Overview" Project discussion Data Centers overview Fat Tree paper discussion CSE
Centec s SDN Switch Built from the Ground Up to Deliver an Optimal Virtual Private Cloud Table of Contents Virtualization Fueling New Possibilities Virtual Private Cloud Offerings... 2 Current Approaches
What is SDN? And Why Should I Care? Jim Metzler Vice President Ashton Metzler & Associates 1 Goals of the Presentation 1. Define/describe SDN 2. Identify the drivers and inhibitors of SDN 3. Identify what
Datacenter architectures Paolo Giaccone Notes for the class on Router and Switch Architectures Politecnico di Torino January 205 Outline What is a data center 2 Datacenter traffic 3 Routing and addressing
SOFTWARE-DEFINED NETWORKING AND OPENFLOW Eric Choi < firstname.lastname@example.org> Senior Manager, Service Provider Business Unit, APJ 2012 Brocade Communications Systems, Inc. EPF 7 2012/09/17 Software-Defined Networking
White Paper Juniper Networks Solutions for VMware NSX Enabling Businesses to Deploy Virtualized Data Center Environments Copyright 2013, Juniper Networks, Inc. 1 Table of Contents Executive Summary...3
Executive Summary: Boosting Business Agility through Software-defined Networking Completing the last mile of virtualization Introduction Businesses have gained significant value from virtualizing server
Cloud Fabric Huawei Cloud Fabric-Cloud Connect Data Center Solution HUAWEI TECHNOLOGIES CO.,LTD. Huawei Cloud Fabric - Cloud Connect Data Center Solution Enable Data Center Networks to Be More Agile for
Virtualization, SDN and NFV HOW DO THEY FIT TOGETHER? Traditional networks lack the flexibility to keep pace with dynamic computing and storage needs of today s data centers. In order to implement changes,
November 2015 Software Defined Environments 2015 Cloud Lecture, University of Stuttgart Jochen Breh, Director Architecture & Consulting Cognizant Global Technology Office Agenda Introduction New Requirements
Avoiding Network Polarization and Increasing Visibility in Cloud Networks Using Broadcom Smart- Hash Technology Sujal Das Product Marketing Director Network Switching Karthik Mandakolathur Sr Product Line
SDN CONTROLLER IN VIRTUAL DATA CENTER Emil Gągała PLNOG, 30.09.2013, Kraków INSTEAD OF AGENDA 2 Copyright 2013 Juniper Networks, Inc. www.juniper.net ACKLOWLEDGEMENTS Many thanks to Bruno Rijsman for his
Testing Software Defined Network (SDN) For Data Center and Cloud VERYX TECHNOLOGIES Table of Contents Introduction... 1 SDN - An Overview... 2 SDN: Solution Layers and its Key Requirements to be validated...
Open Source Networking for Cloud Data Centers Gaetano Borgione Distinguished Engineer @ PLUMgrid April 2015 1 Agenda Open Source Clouds with OpenStack Building Blocks of Cloud Networking Tenant Networks
CLOUD NETWORKING THE NEXT CHAPTER FLORIN BALUS COMMON APPLICATION VIEW OF THE NETWORK Fallacies of Distributed Computing 1. The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4. The
Georg Ochs, Smart Cloud Orchestrator (email@example.com) Software Defined Network (SDN) University of Stuttgart Cloud Course Fall 2013 Agenda Introduction SDN Components Openstack and SDN Example Scenario
Network Virtualization What is Network Virtualization? Abstraction of the physical network Support for multiple logical networks running on a common shared physical substrate A container of network services
RIDE THE SDN AND CLOUD WAVE WITH CONTRAIL Pascal Geenens CONSULTING ENGINEER, JUNIPER NETWORKS firstname.lastname@example.org BUSINESS AGILITY Need to create and deliver new revenue opportunities faster Services
OVERLAYING VIRTUALIZED LAYER 2 NETWORKS OVER LAYER 3 NETWORKS Matt Eclavea (email@example.com) Senior Solutions Architect, Brocade Communications Inc. Jim Allen (firstname.lastname@example.org) Senior Architect, Limelight
VXLAN Extending Networking to Fit the Cloud Kamau WangŨ H Ũ Kamau Wangũhgũ is a Consulting Architect at VMware and a member of the Global Technical Service, Center of Excellence group. Kamau s focus at
Simplify Your Data Center Network to Improve Performance and Decrease Costs Summary Traditional data center networks are struggling to keep up with new computing requirements. Network architects should
Switching Fabric Designs for Data Centers David Klebanov Technical Solutions Architect, Cisco Systems email@example.com @DavidKlebanov 1 Agenda Data Center Fabric Design Principles and Industry Trends
Pluribus Netvisor Solution Brief Freedom Architecture Overview The Pluribus Freedom architecture presents a unique combination of switch, compute, storage and bare- metal hypervisor OS technologies, and
WHITE PAPER October 2013 ConnectX -3 Pro: Solving the NVGRE Performance Challenge Objective...1 Background: The Need for Virtualized Overlay Networks...1 NVGRE Technology...2 NVGRE s Hidden Challenge...3
Cloud Computing and the Internet Conferenza GARR 2010 Cloud Computing The current buzzword ;-) Your computing is in the cloud! Provide computing as a utility Similar to Electricity, Water, Phone service,
EVOLVING ENTERPRISE NETWORKS WITH SPB-M APPLICATION NOTE EXECUTIVE SUMMARY Enterprise network managers are being forced to do more with less. Their networks are growing in size and complexity. They need
Cloud Networking Disruption with Software Defined Network Virtualization Ali Khayam In the next one hour Let s discuss two disruptive new paradigms in the world of networking: Network Virtualization Software
Volume 5, Issue 1, January 2015 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Survey on Load
WHITE PAPER Software-Defined Networks Powered by VellOS Agile, Flexible Networking for Distributed Applications Vello s SDN enables a low-latency, programmable solution resulting in a faster and more flexible
PRAMAK 1 Optimizing Data Center Networks for Cloud Computing Data Center networks have evolved over time as the nature of computing changed. They evolved to handle the computing models based on main-frames,
SDN AND SECURITY: Why Take Over the s When You Can Take Over the Network SESSION ID: TECH0R03 Robert M. Hinden Check Point Fellow Check Point Software What are the SDN Security Challenges? Vulnerability
Carrier/WAN SDN SDN Optimized MPLS Demo Problem Statement! Service Providers around the world have large investments in highly sophisticated and feature rich IP/MPLS network infrastructures for providing
Network performance in virtual infrastructures A closer look at Amazon EC2 Alexandru-Dorin GIURGIU University of Amsterdam System and Network Engineering Master 03 February 2010 Coordinators: Paola Grosso
Improving the Scalability of Data Center Networks with Traffic-aware Virtual Machine Placement Xiaoqiao Meng, Vasileios Pappas, Li Zhang IBM T.J. Watson Research Center Presented by: Payman Khani Overview:
Software-Defined Networking for the Data Center Dr. Peer Hasselmeyer NEC Laboratories Europe NW Technology Can t Cope with Current Needs We still use old technology... but we just pimp it To make it suitable
UNIFI@ECTI-CON 2013 (May 14 th 17 th 2013, Krabi, Thailand) Combined Smart Sleeping and Power Scaling for Energy Efficiency in Green Data Center Networks Nguyen Huu Thanh Department of Communications Engineering
Non-blocking Switching in the Cloud Computing Era Contents 1 Foreword... 3 2 Networks Must Go With the Flow in the Cloud Computing Era... 3 3 Fat-tree Architecture Achieves a Non-blocking Data Center Network...
CS6204 Advanced Topics in Networking Assoc Prof. Chan Mun Choon School of Computing National University of Singapore Aug 14, 2015 CS6204 Lecturer Chan Mun Choon Office: COM2, #04-17 Email: firstname.lastname@example.org
Networking Issues For Big Data. Raj Jain Washington University in Saint Louis Saint Louis, MO 63130 Jain@cse.wustl.edu These slides and audio/video recordings of this class lecture are at: http://www.cse.wustl.edu/~jain/cse570-13/
Optical interconnects in data centers Optical Communication Systems and Networks: From Data Centers to Broadband Access to Core Transport Networks 9 May 2014 Christoforos Kachris Athens Information Technology
Institute of Computer and Communication Network Engineering Institute of Computer and Communication Network Engineering Communication Networks Software Defined Networking (SDN) Prof. Dr. Admela Jukan Dr.
SOFTWARE DEFINED NETWORKING Bringing Networks to the Cloud Brendan Hayes DIRECTOR, SDN MARKETING AGENDA Market trends and Juniper s SDN strategy Network virtualization evolution Juniper s SDN technology
Radhika Niranjan Mysore, Andreas Pamboris, Nathan Farrington, Nelson Huang, Pardis Miri, Sivasankar Radhakrishnan, Vikram Subramanya and Amin Vahdat 1 PortLand In A Nutshell PortLand is a single logical
Mobile Cloud Computing Lecture 02b Cloud Computing II 吳 秀 陽 Shiow-yang Wu T. Sridhar. Cloud Computing A Primer, Part 2: Infrastructure and Implementation Topics. The Internet Protocol Journal, Volume 12,
White Paper on NETWORK VIRTUALIZATION INDEX 1. Introduction 2. Key features of Network Virtualization 3. Benefits of Network Virtualization 4. Architecture of Network Virtualization 5. Implementation Examples
Mobile Cloud Computing Lecture 02a Cloud Computing I 吳 秀 陽 Shiow-yang Wu What is Cloud Computing? Computing with cloud? Mobile Cloud Computing Cloud Computing I 2 Note 1 What is Cloud Computing? Walking
4th International Conference on Machinery, Materials and Computing Technology (ICMMCT 2016) Research on Cloud Computing Network Architecture Based on SDN Technology Weibo Li wuhan textile university, Wuhan,
AIN: A Blueprint for an All-IP Data Center Network Vasileios Pappas Hani Jamjoom Dan Williams IBM T. J. Watson Research Center, Yorktown Heights, NY Abstract With both Ethernet and IP powering Data Center
Ethernet-based Software Defined Network (SDN) Cloud Computing Research Center for Mobile Applications (CCMA), ITRI 雲 端 運 算 行 動 應 用 研 究 中 心 1 SDN Introduction Decoupling of control plane from data plane
Ahmad Zamer, Brocade SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations
BROCADE SOFTWARE DEFINED NETWORKING: INDUSTRY INVOLVEMENT Rajesh Dhople Brocade Communications Systems, Inc. email@example.com 2012 Brocade Communications Systems, Inc. 1 Why can t you do these things
Virtualization and SDN lications 2 Virtualization Sharing physical hardware or software resources by multiple users and/or use cases Examples system shares physical hardware resources Virtual machine shares
Introduction The past two decades have seen dramatic shifts in data center design. As application complexity grew, server sprawl pushed out the walls of the data center, expanding both the physical square
Network Virtualization for the Enterprise Data Center Guido Appenzeller Open Networking Summit October 2011 THE ENTERPRISE DATA CENTER! Major Trends change Enterprise Data Center Networking Trends in the
OpenFlow and Onix Bowei Xu firstname.lastname@example.org  McKeown et al., "OpenFlow: Enabling Innovation in Campus Networks," ACM SIGCOMM CCR, 38(2):69-74, Apr. 2008.  Koponen et al., "Onix: a Distributed Control
SDN Applications in Today s Data Center Harry Petty Director Data Center & Cloud Networking Cisco Systems, Inc. Santa Clara, CA USA October 2013 1 Customer Insights: Research/ Academia OpenFlow/SDN components
Introduction to Software Defined Networking (SDN) and how it will change the inside of your DataCentre Wilfried van Haeren CTO Edgeworx Solutions Inc. www.edgeworx.solutions Topics Intro Edgeworx Past-Present-Future
Dynamic Resource management with VM layer and Resource prediction algorithms in Cloud Architecture 1 Shaik Fayaz, 2 Dr.V.N.Srinivasu, 3 Tata Venkateswarlu #1 M.Tech (CSE) from P.N.C & Vijai Institute of
Extreme Networks: Building Cloud-Scale Networks Using Open Fabric Architectures A SOLUTION WHITE PAPER WHITE PAPER Building Cloud- Scale Networks Abstract TABLE OF CONTENTS Introduction 2 Open Fabric-Based
Network Virtualization and Application Delivery Using Software Defined Networking Project Leader: Subharthi Paul Washington University in Saint Louis Saint Louis, MO 63130 Jain@cse.wustl.edu Keynote at
Dynamic Resource Allocation in Software Defined and Virtual Networks: A Comparative Analysis Felipe Augusto Nunes de Oliveira - GRR20112021 João Victor Tozatti Risso - GRR20120726 Abstract. The increasing
WHITE PAPER DATA CENTER Multitenancy Options in Brocade VCS Fabrics As cloud environments reach mainstream adoption, achieving scalable network segmentation takes on new urgency to support multitenancy.
Enterprise Data Center Networks Isabelle Guis Big Switch Networks Vice President of Outbound Marketing ONF Market Education Committee Chair 1 This Session Objectives Leave with an understanding of Data
for Service Provider Data Center and IXP Francois Tallet, Cisco Systems 1 : Transparent Interconnection of Lots of Links overview How works designs Conclusion 2 IETF standard for Layer 2 multipathing Driven
Network Technologies for Next-generation Data Centers SDN-VE: Software Defined Networking for Virtual Environment Rami Cohen, IBM Haifa Research Lab September 2013 Data Center Network Defining and deploying
Improving Network Management with Software Defined Networking Hyojoon Kim and Nick Feamster, Georgia Institute of Technology 2013 IEEE Communications Magazine Presented by 101062505 林 瑋 琮 Outline 1. Introduction
Software Defined Networks Damiano Carra Università degli Studi di Verona Dipartimento di Informatica Acknowledgements! Credits Part of the course material is based on slides provided by the following authors