Locality-Sensitive Operators for Parallel Main-Memory Database Clusters
|
|
- Rafe Knight
- 8 years ago
- Views:
Transcription
1 Locality-Sensitive Operators for Parallel Main-Memory Database Clusters Wolf Rödiger, Tobias Mühlbauer, Philipp Unterbrunner*, Angelika Reiser, Alfons Kemper, Thomas Neumann Technische Universität München, *Snowflake Computing, Inc. 1 / 31
2 Scale Out HyPer: High-performance sort in-memory transaction and query processing system group Scale out to process extremely largejoin inputs Aiming at clusters with large mainlineitem memory capacityorders TPC-H Query 12 node 2 node 1 node 3 SWITCH node 4 node 6 node 5 Main-Memory Cluster 2 / 31
3 Running Example (1) Focus on analytical query processing in this talk TPC-H query 12 used as running example Runtime dominated by join orders lineitem lineitem sort group join orders n n TPC-H Query 12 3 / 31
4 Running Example (2) orders lineitem Relations are equally distributed across nodes We make no other assumptions on data distribution Network communication required as tuples join with tuples on remote nodes node 3 node 2 node 1 key priority 1 1-URGENT 2 2-HIGH 3 1-URGENT 4 5-LOW 5 3-MEDIUM 6 1-URGENT 7 2-HIGH 8 1-URGENT 9 1-URGENT 10 2-HIGH 11 3-MEDIUM 12 5-LOW 13 1-URGENT 14 3-MEDIUM 15 1-URGENT 16 3-MEDIUM 17 2-HIGH 18 3-MEDIUM 19 5-LOW 20 1-URGENT 21 2-HIGH key shipmode 1 MAIL 1 MAIL 1 MAIL 2 SHIP 2 MAIL 6 SHIP 6 SHIP 6 SHIP 6 MAIL 10 SHIP 11 MAIL 11 MAIL 13 MAIL 13 MAIL 13 MAIL 13 SHIP 17 MAIL 18 MAIL 18 MAIL 19 SHIP 20 SHIP 4 / 31
5 Scale Out: Network is the Bottleneck 200 Single node: Performance is bound algorithmically Cluster: Network is bottleneck for query processing We propose a novel join algorithm called Neo-Join Goal: Increase local processing to close the performance gap join performance [M tuples/s] distributed local 5 / 31
6 Neo-Join: Network-optimized Join 1. Open Shop Scheduling Efficient network communication 2. Optimal Partition Assignment Increase local processing 3. Selective Broadcast Handle value skew 6 / 31
7 Open Shop Scheduling Efficient network communication 7 / 31
8 Standard Network Model Star topology Nodes are connected to a central switch Fully switched All links can be used simultaneously Fully duplex Nodes can both send and receive at full speed node 1 node 4 node 2 SWITCH node 5 node 3 node 6 8 / 31
9 Bandwidth Sharing node 11 node 22 node 11 bottleneck SWITCH bottleneck node 33 SWITCH node 22 node 33 Simultaneous use of a single link creates a bottleneck Reduces bandwidth by at least a factor of 2 9 / 31
10 Naïve Schedule node node node bandwidth sharing time Node node 21and 3 send 4 to node 1 at 5 the same 1 time Bandwidth node 2 sharing 4 increases 4network duration 1 significantly node time maximum straggler 10 / 31
11 Open Shop Scheduling (1) Avoiding bandwidth sharing translates directly to open shop scheduling Network Transfer: Receivers receive from at most one sender, senders send to at most one receiver Open Shop: Processors perform one task at a time, only one task of a job is processed at a time Open Shop task processor job execution time Network Transfer data transfer receiver sender message size 11 / 31
12 Open Shop Scheduling (2) Bipartite graph of senders and receivers Edge weights represent transfer size Scheduler repeatedly finds perfect matchings Each matching specifies one communication phase Transfers in a phase will never share bandwidth node 1 node 1 sender node 2 node node 2 receiver node 3 12 / 31
13 node 2 node Optimal Schedule bandwidth sharing node node node time maximum straggler Open shop schedule achieves minimal network duration Schedule duration determined by maximum straggler 13 / 31
14 Optimal Partition Assignment Minimize network duration for distributed joins 14 / 31
15 Distributed Join Tuples may join with tuples on remote nodes Repartition and redistribute both relations for local join Tuples will join only with the corresponding partition Using hash, range, radix, or other partitioning scheme In any case: Decide how to assign partitions to nodes O L O1 O2 O3 L1 L2 L3 fragmented redistributed 15 / 31
16 Running Example: Hash Partitioning node 3 node 2 node 1 orders lineitem key priority key shipmode 1 1-URGENT 1 MAIL 2 2-HIGH 1 MAIL 3 1-URGENT 1 MAIL 4 5-LOW 2 SHIP 5 3-MEDIUM 2 MAIL x+2 mod URGENT 6 SHIP HIGH 6 SHIP 8 1-URGENT 6 SHIP P1 P2 P3 9 1-URGENT 6 MAIL 10 2-HIGH 10 SHIP 11 3-MEDIUM 11 MAIL 12 5-LOW 11 MAIL 13 1-URGENT 13 MAIL x+2 mod MEDIUM 13 MAIL URGENT P1 P2 P MEDIUM 13 MAIL 17 2-HIGH 13 SHIP 18 3-MEDIUM 17 MAIL 19 5-LOW 18 MAIL 20 1-URGENT 18 MAIL x+2 mod HIGH 19 SHIP SHIP P1 P2 P3 16 / 31
17 Assign Partitions to Nodes (1) Option 1: Minimize network traffic Assign partition to node that owns its largest part Only the small fragments of a partition sent over the network Schedule with minimal network traffic may have high duration n1 n2 n3 hash partitioning (x mod 3) P1 P2 P3 open shop schedule n1 n2 13 n3 13 h n1 n2 n3 n1 n2 n3 traffic: 26 time: 26 t 17 / 31
18 Assign Partitions to Nodes (2) hash partitioning (x mod 3) n Option 2: Minimize response time: n Query response time is time from request n3 4 2 to result Query response P1 time P2 dominated P3 by network duration To minimize network open shop duration, schedule minimize n1 maximum straggler n2 n n1 n2 n3 hash partitioning (x mod 3) P1 P2 P3 open shop schedule n n n traffic: 26 time: 26 traffic: 28 time: / 31
19 ing one would expect that every its tuples to every other node ( of the tuples from every other plified example, radix partitioning network phase by almost a factor rtitioning. Minimize Maximum Straggler Formalized as mixed-integer linear program ment (Phase 2) Objective function minimizes maximum straggler ribed how to repartition the input he same join key fall into the same partitions are fragmented across gments of one specific partition he same node for joining. This rmine an assignment of partitions network phase duration. st of a node as the number of nodes for the partitions that were end cost is defined as the number ther nodes. Section III-C4 shows ase duration is determined by the d/receive cost. The assignment is ize this maximum cost. ssign a partition to the node that However, this is not optimal in ment for the running example in ned to node 1 even though node 0 Shown to be NP-hard (see paper for proof sketch) In practice fast enough using CPLEX or Gurobi (< 0.5 % overhead for 32 nodes, 200 M tuples each) Partition assignment can optimize any partitioning Equation 3 computes the send cost of node i as the size of all local fragments of partitions that are not assigned to it. Likewise, equation 4 adds the size of remote fragments of partitions that were assigned to node i to the receive cost. MILPs require a linear objective, which minimizing a maximum is not. Fortunately, we can rephrase the objective and instead minimize a new variable w. Additional constraints take care that w assumes the maximum over the send/receive costs: (OPT-ASSIGN) minimize w, subject to p 1 w h ij (1 x ij ) j=0 p 1 w x ij j=0 n 1 1 = i=0 x ij n 1 k=0,i k h kj 0 i < n 0 i < n 0 j < p One can obtain an optimal solution for a specific partition assignment problem (OPT-ASSIGN) by passing the mixed integer linear program to an optimizer such as Microsoft Gurobi 3 or IBM CPLEX 4. These solvers can be linked as a library to create and solve linear programs via API calls / 31
20 Running Example: Locality orders lineitem node 1 key priority 1 1-URGENT 2 2-HIGH 3 1-URGENT 4 5-LOW 5 3-MEDIUM 6 1-URGENT 7 2-HIGH 8 1-URGENT key shipmode 1 MAIL 1 MAIL 1 MAIL 2 SHIP 2 MAIL 6 SHIP 6 SHIP 6 SHIP radix 15 P1 1 P2 P3 node URGENT 10 2-HIGH 11 3-MEDIUM 12 5-LOW 13 1-URGENT 14 3-MEDIUM 15 1-URGENT 6 MAIL 10 SHIP 11 MAIL 11 MAIL 13 MAIL 13 MAIL radix P1 P2 P3 node MEDIUM 17 2-HIGH 18 3-MEDIUM 19 5-LOW 20 1-URGENT 21 2-HIGH 13 MAIL 13 SHIP 17 MAIL 18 MAIL 18 MAIL 19 SHIP 20 SHIP radix P1 2 P2 11 P3 20 / 31
21 Locality Running example exhibits time-of-creation clustering Radix repartitioning on most significant bits retains locality Partition assignment can exploit locality Significantly reduces query response time n1 n2 n3 n1 n1 1 n2 n2 1 1 n3 2 n3 radix partitioning (MSB) P1 P2 P2 P3 P3 open shop schedule traffic: 5 time: 3 21 / 31
22 Selective Broadcast Handle value skew 22 / 31
23 Running Example: Skew orders lineitem key priority key shipmode node URGENT 2 2-HIGH 3 1-URGENT 4 1-URGENT 1 MAIL 2 MAIL 2 MAIL 2 MAIL 2 SHIP 2 SHIP 2 SHIP 6 MAIL x+2 mod O1 L1 O2 L2 O3 L3 node 3 node URGENT 6 2-HIGH 7 3-MEDIUM 8 3-MEDIUM 9 2-HIGH 10 3-MEDIUM 6 MAIL 7 SHIP 8 SHIP 8 SHIP 8 MAIL 8 MAIL 8 MAIL 8 MAIL 8 MAIL 8 MAIL 8 MAIL 8 MAIL 8 MAIL 8 MAIL 8 SHIP 10 MAIL x+2 mod 3 x+2 mod O1 L1 O2 L2 O3 L O1 L1 O2 L2 O3 L3 23 / 31
24 Skew Value skew can lead to some very large partitions Assignment of these partitions increases network duration One may try to balance skewed partitions by partitioning the input into more partitions High skew is still a problem n1 n2 n3 hash partitioning (mod 3) P1 P2 P2 P3 P3 open shop schedule n1 n1 7 2 n2 n2 2 7 n3 1 n3 2 traffic: 21 time: / 31
25 Broadcast O Alternative data redistribution scheme Replicate the smaller relation between all nodes O L O L Larger relation remains fragmented across nodes O broadcast O local join 25 / 31
26 Selective Broadcast Decide per partition whether to assign or broadcast Broadcast partitions with large relation size difference Assign the other partitions taking locality into account Role reversal possible: Broadcast different partitions by different relations n1 n2 n3 n1 n1 n2 n2 n3 n3 hash partitioning (mod 3) O1 L1 O2 L2 L2 O3 O3 L3 L3 open shop schedule traffic: 14 time: 6 26 / 31
27 Evaluation 27 / 31
28 Locality Vary locality from 0 % (uniform distribution) to 100 % (range partitioning) Neo-Join improves join performance from 29 M to 156 M tuples/s (> 500 %) 3 nodes (Core i7, 4 cores, 3.4 GHz, 32 GB RAM), 600 M tuples (64 bit key, 64 bit payload) join performance [tuple/s] Neo-Join Hadoop Hive DBMS-X MySQL Cluster 160 M 120 M 80 M 40 M 0 M 0 % 25 % 50 % 75 % 100 % locality 28 / 31
29 Skew Zipfian distribution models realistic data skew Using more partitions alleviates the problem Selective broadcast actually improves performance for skewed inputs 4 nodes, 400 M tuples Zipf factor s partitions s 24 s 23 s 29s 44s s 23 s 23 s 23 s 33s 16 (SB) 24 s 24 s 23 s 20s 10s 29 / 31
30 TPC-H Results (scale factor 100) Results for three selected TPC-H queries Broadcast outperforms hash for large relation size differences Neo-Join always performs better due to selective broadcast and locality 4 nodes, scale factor 100 execution time [s] hash broadcast Neo-Join Q12 Q14 Q19 TPC-H queries 30 / 31
31 Summary Motivation: Network is the bottleneck for distributed query processing Increase local processing to close the performance gap Contributions: Open Shop Scheduling avoids bandwidth sharing Optimal Partition Assignment minimizes query response time and can exploit locality in the data distribution Selective Broadcast combines repartitioning and broadcast to improve the performance for skewed inputs 31 / 31
Locality-Sensitive Operators for Parallel Main-Memory Database Clusters
Locality-Sensitive Operators for Parallel Main-Memory Database Clusters Wolf Rödiger, Tobias Mühlbauer, Philipp Unterbrunner, Angelika Reiser, Alfons Kemper, Thomas Neumann Technische Universität München
More informationFPGA-based Multithreading for In-Memory Hash Joins
FPGA-based Multithreading for In-Memory Hash Joins Robert J. Halstead, Ildar Absalyamov, Walid A. Najjar, Vassilis J. Tsotras University of California, Riverside Outline Background What are FPGAs Multithreaded
More informationHadoop s Entry into the Traditional Analytical DBMS Market. Daniel Abadi Yale University August 3 rd, 2010
Hadoop s Entry into the Traditional Analytical DBMS Market Daniel Abadi Yale University August 3 rd, 2010 Data, Data, Everywhere Data explosion Web 2.0 more user data More devices that sense data More
More informationLoad Balancing in MapReduce Based on Scalable Cardinality Estimates
Load Balancing in MapReduce Based on Scalable Cardinality Estimates Benjamin Gufler 1, Nikolaus Augsten #, Angelika Reiser 3, Alfons Kemper 4 Technische Universität München Boltzmannstraße 3, 85748 Garching
More informationParallel Databases. Parallel Architectures. Parallelism Terminology 1/4/2015. Increase performance by performing operations in parallel
Parallel Databases Increase performance by performing operations in parallel Parallel Architectures Shared memory Shared disk Shared nothing closely coupled loosely coupled Parallelism Terminology Speedup:
More informationMassively Parallel Sort-Merge Joins in Main Memory Multi-Core Database Systems. Martina-Cezara Albutiu, Alfons Kemper, Thomas Neumann
TUM TECHNISCHE UNIVERSITÄT MÜNCHEN INSTITUT FÜR INFORMATIK Massively Parallel Sort-Merge Joins in Main Memory Multi-Core Database Systems Martina-Cezara Albutiu, Alfons Kemper, Thomas Neumann TUM-I121
More informationCan the Elephants Handle the NoSQL Onslaught?
Can the Elephants Handle the NoSQL Onslaught? Avrilia Floratou, Nikhil Teletia David J. DeWitt, Jignesh M. Patel, Donghui Zhang University of Wisconsin-Madison Microsoft Jim Gray Systems Lab Presented
More informationExploiting Remote Memory Operations to Design Efficient Reconfiguration for Shared Data-Centers over InfiniBand
Exploiting Remote Memory Operations to Design Efficient Reconfiguration for Shared Data-Centers over InfiniBand P. Balaji, K. Vaidyanathan, S. Narravula, K. Savitha, H. W. Jin D. K. Panda Network Based
More informationBeyond the Stars: Revisiting Virtual Cluster Embeddings
Beyond the Stars: Revisiting Virtual Cluster Embeddings Matthias Rost Technische Universität Berlin September 7th, 2015, Télécom-ParisTech Joint work with Carlo Fuerst, Stefan Schmid Published in ACM SIGCOMM
More informationPipelining and load-balancing in parallel joins on distributed machines
NP-PAR 05 p. / Pipelining and load-balancing in parallel joins on distributed machines M. Bamha bamha@lifo.univ-orleans.fr Laboratoire d Informatique Fondamentale d Orléans (France) NP-PAR 05 p. / Plan
More informationBig Data Technology Map-Reduce Motivation: Indexing in Search Engines
Big Data Technology Map-Reduce Motivation: Indexing in Search Engines Edward Bortnikov & Ronny Lempel Yahoo Labs, Haifa Indexing in Search Engines Information Retrieval s two main stages: Indexing process
More informationIntegrating Apache Spark with an Enterprise Data Warehouse
Integrating Apache Spark with an Enterprise Warehouse Dr. Michael Wurst, IBM Corporation Architect Spark/R/Python base Integration, In-base Analytics Dr. Toni Bollinger, IBM Corporation Senior Software
More informationChapter 14: Distributed Operating Systems
Chapter 14: Distributed Operating Systems Chapter 14: Distributed Operating Systems Motivation Types of Distributed Operating Systems Network Structure Network Topology Communication Structure Communication
More informationCollaborative Query Coordination in Community-Driven Data Grids
HPDC '09 Collaborative Query Coordination in Community-Driven Data Grids Tobias Scholl, Angelika Reiser, and Alfons Kemper Department of Computer Science, Technische Universität München Germany Community-Driven
More informationModule 15: Network Structures
Module 15: Network Structures Background Topology Network Types Communication Communication Protocol Robustness Design Strategies 15.1 A Distributed System 15.2 Motivation Resource sharing sharing and
More informationA Comparison of Join Algorithms for Log Processing in MapReduce
A Comparison of Join Algorithms for Log Processing in MapReduce Spyros Blanas, Jignesh M. Patel Computer Sciences Department University of Wisconsin-Madison {sblanas,jignesh}@cs.wisc.edu Vuk Ercegovac,
More informationIn-Memory Columnar Databases HyPer. Arto Kärki University of Helsinki 30.11.2012
In-Memory Columnar Databases HyPer Arto Kärki University of Helsinki 30.11.2012 1 Introduction Columnar Databases Design Choices Data Clustering and Compression Conclusion 2 Introduction The relational
More informationParallel Processing of JOIN Queries in OGSA-DAI
Parallel Processing of JOIN Queries in OGSA-DAI Fan Zhu Aug 21, 2009 MSc in High Performance Computing The University of Edinburgh Year of Presentation: 2009 Abstract JOIN Query is the most important and
More informationScalable Data Analysis in R. Lee E. Edlefsen Chief Scientist UserR! 2011
Scalable Data Analysis in R Lee E. Edlefsen Chief Scientist UserR! 2011 1 Introduction Our ability to collect and store data has rapidly been outpacing our ability to analyze it We need scalable data analysis
More informationChapter 16: Distributed Operating Systems
Module 16: Distributed ib System Structure, Silberschatz, Galvin and Gagne 2009 Chapter 16: Distributed Operating Systems Motivation Types of Network-Based Operating Systems Network Structure Network Topology
More informationOperating System Concepts. Operating System 資 訊 工 程 學 系 袁 賢 銘 老 師
Lecture 7: Distributed Operating Systems A Distributed System 7.2 Resource sharing Motivation sharing and printing files at remote sites processing information in a distributed database using remote specialized
More informationReport: Declarative Machine Learning on MapReduce (SystemML)
Report: Declarative Machine Learning on MapReduce (SystemML) Jessica Falk ETH-ID 11-947-512 May 28, 2014 1 Introduction SystemML is a system used to execute machine learning (ML) algorithms in HaDoop,
More informationHigh-performance local search for planning maintenance of EDF nuclear park
High-performance local search for planning maintenance of EDF nuclear park Frédéric Gardi Karim Nouioua Bouygues e-lab, Paris fgardi@bouygues.com Laboratoire d'informatique Fondamentale - CNRS UMR 6166,
More informationTopological Properties
Advanced Computer Architecture Topological Properties Routing Distance: Number of links on route Node degree: Number of channels per node Network diameter: Longest minimum routing distance between any
More informationAccelerating Enterprise Applications and Reducing TCO with SanDisk ZetaScale Software
WHITEPAPER Accelerating Enterprise Applications and Reducing TCO with SanDisk ZetaScale Software SanDisk ZetaScale software unlocks the full benefits of flash for In-Memory Compute and NoSQL applications
More informationA Novel Cloud Based Elastic Framework for Big Data Preprocessing
School of Systems Engineering A Novel Cloud Based Elastic Framework for Big Data Preprocessing Omer Dawelbeit and Rachel McCrindle October 21, 2014 University of Reading 2008 www.reading.ac.uk Overview
More informationParallel Computing. Benson Muite. benson.muite@ut.ee http://math.ut.ee/ benson. https://courses.cs.ut.ee/2014/paralleel/fall/main/homepage
Parallel Computing Benson Muite benson.muite@ut.ee http://math.ut.ee/ benson https://courses.cs.ut.ee/2014/paralleel/fall/main/homepage 3 November 2014 Hadoop, Review Hadoop Hadoop History Hadoop Framework
More informationAchieving Nanosecond Latency Between Applications with IPC Shared Memory Messaging
Achieving Nanosecond Latency Between Applications with IPC Shared Memory Messaging In some markets and scenarios where competitive advantage is all about speed, speed is measured in micro- and even nano-seconds.
More informationComponents: Interconnect Page 1 of 18
Components: Interconnect Page 1 of 18 PE to PE interconnect: The most expensive supercomputer component Possible implementations: FULL INTERCONNECTION: The ideal Usually not attainable Each PE has a direct
More informationBig Data and Apache Hadoop s MapReduce
Big Data and Apache Hadoop s MapReduce Michael Hahsler Computer Science and Engineering Southern Methodist University January 23, 2012 Michael Hahsler (SMU/CSE) Hadoop/MapReduce January 23, 2012 1 / 23
More informationGeoGrid Project and Experiences with Hadoop
GeoGrid Project and Experiences with Hadoop Gong Zhang and Ling Liu Distributed Data Intensive Systems Lab (DiSL) Center for Experimental Computer Systems Research (CERCS) Georgia Institute of Technology
More informationAdvances in Virtualization In Support of In-Memory Big Data Applications
9/29/15 HPTS 2015 1 Advances in Virtualization In Support of In-Memory Big Data Applications SCALE SIMPLIFY OPTIMIZE EVOLVE Ike Nassi Ike.nassi@tidalscale.com 9/29/15 HPTS 2015 2 What is the Problem We
More informationCapacity Planning Guide for Adobe LiveCycle Data Services 2.6
White Paper Capacity Planning Guide for Adobe LiveCycle Data Services 2.6 Create applications that can deliver thousands of messages per second to thousands of end users simultaneously Table of contents
More informationSystem Interconnect Architectures. Goals and Analysis. Network Properties and Routing. Terminology - 2. Terminology - 1
System Interconnect Architectures CSCI 8150 Advanced Computer Architecture Hwang, Chapter 2 Program and Network Properties 2.4 System Interconnect Architectures Direct networks for static connections Indirect
More informationUsing diversification, communication and parallelism to solve mixed-integer linear programs
Using diversification, communication and parallelism to solve mixed-integer linear programs R. Carvajal a,, S. Ahmed a, G. Nemhauser a, K. Furman b, V. Goel c, Y. Shao c a Industrial and Systems Engineering,
More informationQuery Optimization in Cloud Environment
Query Optimization in Cloud Environment Cindy Chen Computer Science Department University of Massachusetts Lowell May 31, 2014 OUTLINE Introduction Our Approach Performance Evaluation Conclusion and Future
More informationMAPREDUCE Programming Model
CS 2510 COMPUTER OPERATING SYSTEMS Cloud Computing MAPREDUCE Dr. Taieb Znati Computer Science Department University of Pittsburgh MAPREDUCE Programming Model Scaling Data Intensive Application MapReduce
More informationMapReduce Jeffrey Dean and Sanjay Ghemawat. Background context
MapReduce Jeffrey Dean and Sanjay Ghemawat Background context BIG DATA!! o Large-scale services generate huge volumes of data: logs, crawls, user databases, web site content, etc. o Very useful to be able
More informationHigh Performance Spatial Queries and Analytics for Spatial Big Data. Fusheng Wang. Department of Biomedical Informatics Emory University
High Performance Spatial Queries and Analytics for Spatial Big Data Fusheng Wang Department of Biomedical Informatics Emory University Introduction Spatial Big Data Geo-crowdsourcing:OpenStreetMap Remote
More informationBig Data Analytics with MapReduce VL Implementierung von Datenbanksystemen 05-Feb-13
Big Data Analytics with MapReduce VL Implementierung von Datenbanksystemen 05-Feb-13 Astrid Rheinländer Wissensmanagement in der Bioinformatik What is Big Data? collection of data sets so large and complex
More informationCapacity Planning for Virtualized Servers 1
Capacity Planning for Virtualized Servers 1 Martin Bichler, Thomas Setzer, Benjamin Speitkamp Department of Informatics, TU München 85748 Garching/Munich, Germany (bichler setzer benjamin.speitkamp)@in.tum.de
More informationDistributed Architecture of Oracle Database In-memory
Distributed Architecture of Oracle Database In-memory Niloy Mukherjee, Shasank Chavan, Maria Colgan, Dinesh Das, Mike Gleeson, Sanket Hase, Allison Holloway, Hui Jin, Jesse Kamp, Kartik Kulkarni, Tirthankar
More informationUnderstanding the Benefits of IBM SPSS Statistics Server
IBM SPSS Statistics Server Understanding the Benefits of IBM SPSS Statistics Server Contents: 1 Introduction 2 Performance 101: Understanding the drivers of better performance 3 Why performance is faster
More informationIndex Terms : Load rebalance, distributed file systems, clouds, movement cost, load imbalance, chunk.
Load Rebalancing for Distributed File Systems in Clouds. Smita Salunkhe, S. S. Sannakki Department of Computer Science and Engineering KLS Gogte Institute of Technology, Belgaum, Karnataka, India Affiliated
More informationVMWARE WHITE PAPER 1
1 VMWARE WHITE PAPER Introduction This paper outlines the considerations that affect network throughput. The paper examines the applications deployed on top of a virtual infrastructure and discusses the
More informationBringing Big Data Modelling into the Hands of Domain Experts
Bringing Big Data Modelling into the Hands of Domain Experts David Willingham Senior Application Engineer MathWorks david.willingham@mathworks.com.au 2015 The MathWorks, Inc. 1 Data is the sword of the
More informationBig Data Processing with Google s MapReduce. Alexandru Costan
1 Big Data Processing with Google s MapReduce Alexandru Costan Outline Motivation MapReduce programming model Examples MapReduce system architecture Limitations Extensions 2 Motivation Big Data @Google:
More informationArchitectures for Big Data Analytics A database perspective
Architectures for Big Data Analytics A database perspective Fernando Velez Director of Product Management Enterprise Information Management, SAP June 2013 Outline Big Data Analytics Requirements Spectrum
More informationTHE PROBLEM WORMS (1) WORMS (2) THE PROBLEM OF WORM PROPAGATION/PREVENTION THE MINIMUM VERTEX COVER PROBLEM
1 THE PROBLEM OF WORM PROPAGATION/PREVENTION I.E. THE MINIMUM VERTEX COVER PROBLEM Prof. Tiziana Calamoneri Network Algorithms A.y. 2014/15 2 THE PROBLEM WORMS (1)! A computer worm is a standalone malware
More informationMapReduce and Distributed Data Analysis. Sergei Vassilvitskii Google Research
MapReduce and Distributed Data Analysis Google Research 1 Dealing With Massive Data 2 2 Dealing With Massive Data Polynomial Memory Sublinear RAM Sketches External Memory Property Testing 3 3 Dealing With
More informationA Survey of Cloud Computing Guanfeng Octides
A Survey of Cloud Computing Guanfeng Nov 7, 2010 Abstract The principal service provided by cloud computing is that underlying infrastructure, which often consists of compute resources like storage, processors,
More informationChapter 15: Distributed Structures. Topology
1 1 Chapter 15: Distributed Structures Topology Network Types Operating System Concepts 15.1 Topology Sites in the system can be physically connected in a variety of ways; they are compared with respect
More informationBinary search tree with SIMD bandwidth optimization using SSE
Binary search tree with SIMD bandwidth optimization using SSE Bowen Zhang, Xinwei Li 1.ABSTRACT In-memory tree structured index search is a fundamental database operation. Modern processors provide tremendous
More informationDISTRIBUTED AND PARALLELL DATABASE
DISTRIBUTED AND PARALLELL DATABASE SYSTEMS Tore Risch Uppsala Database Laboratory Department of Information Technology Uppsala University Sweden http://user.it.uu.se/~torer PAGE 1 What is a Distributed
More informationNew Approach of Computing Data Cubes in Data Warehousing
International Journal of Information & Computation Technology. ISSN 0974-2239 Volume 4, Number 14 (2014), pp. 1411-1417 International Research Publications House http://www. irphouse.com New Approach of
More informationDeveloping MapReduce Programs
Cloud Computing Developing MapReduce Programs Dell Zhang Birkbeck, University of London 2015/16 MapReduce Algorithm Design MapReduce: Recap Programmers must specify two functions: map (k, v) * Takes
More informationResource Allocation Schemes for Gang Scheduling
Resource Allocation Schemes for Gang Scheduling B. B. Zhou School of Computing and Mathematics Deakin University Geelong, VIC 327, Australia D. Walsh R. P. Brent Department of Computer Science Australian
More informationPoznań University of Technology
Poznań University of Technology Algorithms to Mitigate Partition Skew in MapReduce Applications J.Berlińska, M.Drozdowski Research Report RA-01/15 2015 Institute of Computing Science, Piotrowo 2, 60-965
More informationBigData. An Overview of Several Approaches. David Mera 16/12/2013. Masaryk University Brno, Czech Republic
BigData An Overview of Several Approaches David Mera Masaryk University Brno, Czech Republic 16/12/2013 Table of Contents 1 Introduction 2 Terminology 3 Approaches focused on batch data processing MapReduce-Hadoop
More informationAdaptive Tolerance Algorithm for Distributed Top-K Monitoring with Bandwidth Constraints
Adaptive Tolerance Algorithm for Distributed Top-K Monitoring with Bandwidth Constraints Michael Bauer, Srinivasan Ravichandran University of Wisconsin-Madison Department of Computer Sciences {bauer, srini}@cs.wisc.edu
More informationRevoScaleR Speed and Scalability
EXECUTIVE WHITE PAPER RevoScaleR Speed and Scalability By Lee Edlefsen Ph.D., Chief Scientist, Revolution Analytics Abstract RevoScaleR, the Big Data predictive analytics library included with Revolution
More informationPowerful Duo: MapR Big Data Analytics with Cisco ACI Network Switches
Powerful Duo: MapR Big Data Analytics with Cisco ACI Network Switches Introduction For companies that want to quickly gain insights into or opportunities from big data - the dramatic volume growth in corporate
More informationMizan: A System for Dynamic Load Balancing in Large-scale Graph Processing
/35 Mizan: A System for Dynamic Load Balancing in Large-scale Graph Processing Zuhair Khayyat 1 Karim Awara 1 Amani Alonazi 1 Hani Jamjoom 2 Dan Williams 2 Panos Kalnis 1 1 King Abdullah University of
More informationManagement & Analysis of Big Data in Zenith Team
Management & Analysis of Big Data in Zenith Team Zenith Team, INRIA & LIRMM Outline Introduction to MapReduce Dealing with Data Skew in Big Data Processing Data Partitioning for MapReduce Frequent Sequence
More informationCSE-E5430 Scalable Cloud Computing Lecture 2
CSE-E5430 Scalable Cloud Computing Lecture 2 Keijo Heljanko Department of Computer Science School of Science Aalto University keijo.heljanko@aalto.fi 14.9-2015 1/36 Google MapReduce A scalable batch processing
More informationTowards Energy-Efficient Database Cluster Design
Towards Energy-Efficient Database Cluster Design Willis Lang University of Wisconsin wlang@cs.wisc.edu Mehul A. Shah Nou Data mehul@noudata.com Stavros Harizopoulos Nou Data stavros@noudata.com Dimitris
More informationGraph Database Proof of Concept Report
Objectivity, Inc. Graph Database Proof of Concept Report Managing The Internet of Things Table of Contents Executive Summary 3 Background 3 Proof of Concept 4 Dataset 4 Process 4 Query Catalog 4 Environment
More informationOverview on Graph Datastores and Graph Computing Systems. -- Litao Deng (Cloud Computing Group) 06-08-2012
Overview on Graph Datastores and Graph Computing Systems -- Litao Deng (Cloud Computing Group) 06-08-2012 Graph - Everywhere 1: Friendship Graph 2: Food Graph 3: Internet Graph Most of the relationships
More informationDeployment Planning Guide
Deployment Planning Guide Community 1.5.0 release The purpose of this document is to educate the user about the different strategies that can be adopted to optimize the usage of Jumbune on Hadoop and also
More informationA Comparison of General Approaches to Multiprocessor Scheduling
A Comparison of General Approaches to Multiprocessor Scheduling Jing-Chiou Liou AT&T Laboratories Middletown, NJ 0778, USA jing@jolt.mt.att.com Michael A. Palis Department of Computer Science Rutgers University
More informationFAST 11. Yongseok Oh <ysoh@uos.ac.kr> University of Seoul. Mobile Embedded System Laboratory
CAFTL: A Content-Aware Flash Translation Layer Enhancing the Lifespan of flash Memory based Solid State Drives FAST 11 Yongseok Oh University of Seoul Mobile Embedded System Laboratory
More informationBigMemory & Hybris : Working together to improve the e-commerce customer experience
& Hybris : Working together to improve the e-commerce customer experience TABLE OF CONTENTS 1 Introduction 1 Why in-memory? 2 Why is in-memory Important for an e-commerce environment? 2 Why? 3 How does
More informationCollaborative Query Coordination in Community-Driven Data Grids
HPDC '09 Collaborative Query Coordination in Community-Driven Data Grids Tobias Scholl, Angelika Reiser, and Alfons Kemper Department of Computer Science, Germany Community-Driven Data Grids (HiSbase)
More informationA1 and FARM scalable graph database on top of a transactional memory layer
A1 and FARM scalable graph database on top of a transactional memory layer Miguel Castro, Aleksandar Dragojević, Dushyanth Narayanan, Ed Nightingale, Alex Shamis Richie Khanna, Matt Renzelmann Chiranjeeb
More informationResearch Statement Immanuel Trummer www.itrummer.org
Research Statement Immanuel Trummer www.itrummer.org We are collecting data at unprecedented rates. This data contains valuable insights, but we need complex analytics to extract them. My research focuses
More informationSQL Server 2005 Features Comparison
Page 1 of 10 Quick Links Home Worldwide Search Microsoft.com for: Go : Home Product Information How to Buy Editions Learning Downloads Support Partners Technologies Solutions Community Previous Versions
More informationScalable Architecture on Amazon AWS Cloud
Scalable Architecture on Amazon AWS Cloud Kalpak Shah Founder & CEO, Clogeny Technologies kalpak@clogeny.com 1 * http://www.rightscale.com/products/cloud-computing-uses/scalable-website.php 2 Architect
More informationMaximizing Hadoop Performance and Storage Capacity with AltraHD TM
Maximizing Hadoop Performance and Storage Capacity with AltraHD TM Executive Summary The explosion of internet data, driven in large part by the growth of more and more powerful mobile devices, has created
More informationEfficient Processing of Data Warehousing Queries in a Split Execution Environment
Efficient Processing of Data Warehousing Queries in a Split Execution Environment Kamil Bajda-Pawlikowski 1+2, Daniel J. Abadi 1+2, Avi Silberschatz 2, Erik Paulson 3 1 Hadapt Inc., 2 Yale University,
More informationDistributed Aggregation in Cloud Databases. By: Aparna Tiwari tiwaria@umail.iu.edu
Distributed Aggregation in Cloud Databases By: Aparna Tiwari tiwaria@umail.iu.edu ABSTRACT Data intensive applications rely heavily on aggregation functions for extraction of data according to user requirements.
More informationDatabase Scalability and Oracle 12c
Database Scalability and Oracle 12c Marcelle Kratochvil CTO Piction ACE Director All Data/Any Data marcelle@piction.com Warning I will be covering topics and saying things that will cause a rethink in
More informationHow To Scale Out Of A Nosql Database
Firebird meets NoSQL (Apache HBase) Case Study Firebird Conference 2011 Luxembourg 25.11.2011 26.11.2011 Thomas Steinmaurer DI +43 7236 3343 896 thomas.steinmaurer@scch.at www.scch.at Michael Zwick DI
More informationTackling Big Data with MATLAB Adam Filion Application Engineer MathWorks, Inc.
Tackling Big Data with MATLAB Adam Filion Application Engineer MathWorks, Inc. 2015 The MathWorks, Inc. 1 Challenges of Big Data Any collection of data sets so large and complex that it becomes difficult
More informationPerformance Analysis of Cloud Relational Database Services
Performance Analysis of Cloud Relational Database Services Jialin Li lijl@cs.washington.edu Naveen Kr. Sharma naveenks@cs.washington.edu June 7, 203 Adriana Szekeres aaazs@cs.washington.edu Introduction
More informationCloud Computing on Amazon's EC2
Technical Report Number CSSE10-04 1. Introduction to Amazon s EC2 Brandon K Maharrey maharbk@auburn.edu COMP 6330 Parallel and Distributed Computing Spring 2009 Final Project Technical Report Cloud Computing
More information18-731 Midterm. Name: Andrew user id:
18-731 Midterm 6 March 2008 Name: Andrew user id: Scores: Problem 0 (10 points): Problem 1 (10 points): Problem 2 (15 points): Problem 3 (10 points): Problem 4 (20 points): Problem 5 (10 points): Problem
More informationEnergy Efficient MapReduce
Energy Efficient MapReduce Motivation: Energy consumption is an important aspect of datacenters efficiency, the total power consumption in the united states has doubled from 2000 to 2005, representing
More informationWhy compute in parallel? Cloud computing. Big Data 11/29/15. Introduction to Data Management CSE 344. Science is Facing a Data Deluge!
Why compute in parallel? Introduction to Data Management CSE 344 Lectures 23 and 24 Parallel Databases Most processors have multiple cores Can run multiple jobs simultaneously Natural extension of txn
More informationHadoop. http://hadoop.apache.org/ Sunday, November 25, 12
Hadoop http://hadoop.apache.org/ What Is Apache Hadoop? The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using
More informationFacebook s Petabyte Scale Data Warehouse using Hive and Hadoop
Facebook s Petabyte Scale Data Warehouse using Hive and Hadoop Why Another Data Warehousing System? Data, data and more data 200GB per day in March 2008 12+TB(compressed) raw data per day today Trends
More informationUse of Hadoop File System for Nuclear Physics Analyses in STAR
1 Use of Hadoop File System for Nuclear Physics Analyses in STAR EVAN SANGALINE UC DAVIS Motivations 2 Data storage a key component of analysis requirements Transmission and storage across diverse resources
More informationStatistical Learning Theory Meets Big Data
Statistical Learning Theory Meets Big Data Randomized algorithms for frequent itemsets Eli Upfal Brown University Data, data, data In God we trust, all others (must) bring data Prof. W.E. Deming, Statistician,
More informationChallenges for Data Driven Systems
Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Quick History of Data Management 4000 B C Manual recording From tablets to papyrus to paper A. Payberah 2014 2
More informationA Review of Customized Dynamic Load Balancing for a Network of Workstations
A Review of Customized Dynamic Load Balancing for a Network of Workstations Taken from work done by: Mohammed Javeed Zaki, Wei Li, Srinivasan Parthasarathy Computer Science Department, University of Rochester
More informationIntroduction to DISC and Hadoop
Introduction to DISC and Hadoop Alice E. Fischer April 24, 2009 Alice E. Fischer DISC... 1/20 1 2 History Hadoop provides a three-layer paradigm Alice E. Fischer DISC... 2/20 Parallel Computing Past and
More informationMapReduce and Hadoop. Aaron Birkland Cornell Center for Advanced Computing. January 2012
MapReduce and Hadoop Aaron Birkland Cornell Center for Advanced Computing January 2012 Motivation Simple programming model for Big Data Distributed, parallel but hides this Established success at petabyte
More informationA Study on Workload Imbalance Issues in Data Intensive Distributed Computing
A Study on Workload Imbalance Issues in Data Intensive Distributed Computing Sven Groot 1, Kazuo Goda 1, and Masaru Kitsuregawa 1 University of Tokyo, 4-6-1 Komaba, Meguro-ku, Tokyo 153-8505, Japan Abstract.
More informationStreaming Similarity Search over one Billion Tweets using Parallel Locality-Sensitive Hashing
Streaming Similarity Search over one Billion Tweets using Parallel Locality-Sensitive Hashing Narayanan Sundaram, Aizana Turmukhametova, Nadathur Satish, Todd Mostak, Piotr Indyk, Samuel Madden and Pradeep
More informationIn-Memory Databases Algorithms and Data Structures on Modern Hardware. Martin Faust David Schwalb Jens Krüger Jürgen Müller
In-Memory Databases Algorithms and Data Structures on Modern Hardware Martin Faust David Schwalb Jens Krüger Jürgen Müller The Free Lunch Is Over 2 Number of transistors per CPU increases Clock frequency
More informationICONICS Choosing the Correct Edition of MS SQL Server
Description: This application note aims to assist you in choosing the right edition of Microsoft SQL server for your ICONICS applications. OS Requirement: XP Win 2000, XP Pro, Server 2003, Vista, Server
More information