HPC in the age of (Big)Data
|
|
- Marvin Corey Lambert
- 8 years ago
- Views:
Transcription
1 HPC in the age of (Big)Data The fusion of supercomputing and Big, Fast Data Analytics Ugo Varetto CRAY EMEA Research Lab
2 Agenda Outlook Requirements Implementation 2
3 Outlook
4 Cray s Vision: The Fusion of Supercomputing and Big, Fast Data Math Models Modeling and simulation augmented with data to provide the highest fidelity virtual reality results Compute Store Data-Intensive Processing High throughput data: capture, processing, retention, and management Analyze Data Models Integration of datasets and math models for search, analysis, predictive modeling and knowledge discovery 4
5 An Enormous Opportunity Scientific Data Historical Data Sensor Data Research & Demographics Social Media Big Data Public Data 3 rd Party Data Enterprise Data Operational Data Transactions 5
6 Data processing: Rising demand 6
7 HPC: New large-scale data-intensive projects Whole system design for Interactive Supercomputing (PCP) 100 Petaflops for data processing Establish platforms to preserve and share data C O M P U T E S T O R E A N A L Y Z E 7
8 HPC: Data Centric Systems Development Accommodate new data centric system architectures Handle data motion highly variable in Size Access patterns Temporal behavior System level co-design required: I/O Compute Interconnect fabric Storage Reliability Software 8
9 Next generation data centric HPC - Implications (Industry) Systems of Insight : Tightly integrated design - modeling simulation loop Interactive analysis, visualization and simulation (Basic Science) Discovery through: Modeling Simulation Analysis 9
10 The traditional scientific discovery workflow Observe Postulate Model Predict Validate 10
11 The emerging scientific discovery workflow Acquire, filter, preprocess Store Insights C O M P U T E S T O R E A N A L Y Z E 11
12 The Fourth Paradigm (?) 1. Theory 2. Experiments 3. Computer Simulations 4. Data-Intensive and Data-Driven Discovery C O M P U T E S T O R E A N A L Y Z E 12
13 The intersection of HPC and BigData Traditional HPC tasks with BigData tools E.g. (Sparse) linear algebra with MLlib + Breeze and jblas Traditional BigData tasks on HPC systems: E.g. SPIDAL (Scalable Parallel Interoperable Data Analytics Library), NSF funded HPC Numerical Libraries BigData BigData on Cray HPC systems (undisclosed project): Standard analytics tools Optimized graph analytics engine myhadoop: Hadoop on standard HPC systems 13
14 Multi-scale Equation-free Modeling Standard modeling: Description of system at fine level Question asked are at macroscopic level Derive macroscopic behavior from microscopic evolution equations Equation Free modeling: 1. Run short bursts of simulation at fine scale 2. Input results from (1) into coarse scale simulation 3. Use results from (2) to initialize next burst of fine scale computation 4. Mapping between fine and coarse scale (w/ dimensionality reduction if needed) 14
15 The fusion of HPC and BigData Commun. Math. Sci. Volume 1, Number 4 (2003), Equation-Free, Coarse-Grained Multiscale Computation: Enabling Macroscopic Simulators to Perform System-Level Analysis - C. William Gear, James M. Hyman, Panagiotis Yannnis G. Kevrekidis, Olof Runborg, and Constantinos Theodoropoulos Coarse time stepper HPC ensemble * * ML? Data-intensive/ Data-driven 15
16 The fusion of HPC and BigData Integrated simulation and data analysis Simulation Amber MD HPC Averaging & Pattern matching BigData 16
17 Requirements
18 HPC and BigData HPC Fast single-node performance Fast interconnect (with RDMA) Homogeneous (w.r.t. software tools, languages) environment Simple job scheduling policies Regular memory access patterns Floating point BigData Reliability Heterogeneous environment Complex task-based scheduling Irregular memory access patterns Integer BigData & HPC: Fast and Scalable I/O 18
19 Benchmarks Berkin Özisikyilmaz, Ramanathan Narayanan, Joseph Zambreno, Gokhan Memik, and Alok N. Choudhary. An architectural characterization study of data mining and bioinformatics work-loads. IISWC, pages 61 70,
20 Compute Intensive vs Data Intensive Synergistic Challenges in Data Intensive Science and Exascale Computing DoE ASCAC Data Subcommittee Report March Figure
21 Implementation
22 Components Storage I/O Network Computing Hardware System software - e.g. job scheduler Software Middleware Libraries Programming languages 22
23 Storage Scalable, fast and reliable storage: Sonexion + DataWarp + TAS DataWarp local (to compute node) file system, additional cache layer TAS = Tiered Adaptive Storage: Transparent access to high latency backup storage (e.g. tapes), files are automatically moved between tape and hard drive Same concept as regular CPU cache 23
24 I/O Lustre parallel file system Parallel I/O libraries available (MPI I/O, HDF5, NetCDF) iobuf for fast serial access Optimal access: Each process accesses a different Object Storage Target (a hard drive) Optimal access for BigData: Use Lustre in place of HDFS Stripe count = number of processes Stripe size = file size / stripe count Access data through Cray-optimized parallel I/O library Consider striping at the directory level, files inherit striping from parent directory 24
25 I/O Explore the use of currently available HPC solutions ADIOS I/O abstraction framework Automatic data distribution from configuration Change I/O method on-the-fly Staging: separation of data access from storage Asynchronous Indexing and querying Tested at scale Java API (accessible from JVM languages) Interface)to)apps)for)descrip/on)of)data)(ADIOS,)etc.)) Feedback) Mul/Bresolu/on) methods) Provenance) Data)Management)Services) Buffering) Data)Compression) methods) )Schedule) Data)Indexing) (FastBit))methods) Workflow))Engine) Run/me)engine) Data)movement) Analysis)Plugins) Plugins)to)the)hybrid)staging)area) Visualiza/on)Plugins) AdiosBbp) IDX) HDF5) pnetcdf) raw )data) Image)data) Parallel)and)Distributed)File)System) Viz.)Client) 25
26 ADIOS aspiration Using the Adaptable I/O System (ADIOS) Joint Facilities User Forum on Data-Intensive Computing June 18, 2014 Norbert Podhorszki 26
27 Network - ARIES Cray s custom interconnect Dragonfly topology Adaptive routing All to all oriented o Chassis with 16 compute blades o 128 Sockets o Inter-Aries communication over backplane o Per-Packet adaptive Routing C O M P U T E S T O R E A N A L Y Z E 27
28 Computing HW vendors working on better support for data intensive workloads e.g. Intel TSX Research on new architectures for data-intensive computing e.g. GoblinCore64, RISC-V ISA extension Re-configurable FPGAs Accelerators (useful for the massively parallel part of the analysis) Data-flow computers (e.g. Maxeler) 28
29 Job scheduling system Sample Taverna workflow for gene identification Source: The Fourth Paradigm book page
30 Job scheduling system Task based Dynamic: allocate resources as needed Dynamic job scheduling solution being tested on Cray Integration of workflow management with job scheduler BigData: MESOS, YARN HPC: Swift/K, Swift/T, integrates with SLURM and PBS 30
31 Software - middleware BigData applications rely on vast and diverse set of runtime systems, storage options, software libraries and programming languages Hard to impossible to support everything through a single system-wide configuration Custom Images (Cray), Virtualization, Docker, OpenStack 31
32 Software libraries Many BigData applications use the same algorithms found in HPC: Eigensolvers Factorization Optimization e.g. (stochastic) Gradient Descent SVD FFT BigData-optimized implementations do exist, e.g.: Efficient Multilevel Eigensolvers with Applications to Data Analysis Tasks - Dan Kushnir, Meirav Galun, and Achi Brandt Recent Development in the Sparse Fourier Transform [A compressed Fourier transform for BigData] - IEEE SIGNAL PROCESSING MAGAZINE [91] SEP Use the right tool for the problem at hand: share libraries between HPC and BigData applications 32
33 Optimizing data access Data-intensive/driven computing can greatly benefit from data-flow (vs control-flow) strategies: Configure the computation and structure of data Let the system optimally distribute the data and run the units of computation Similar to Hadoop, Spark approach Chapel: High-level programming language for parallel distributed computing Transparent handling of data exchange among compute nodes Eiger: Compile/run-time framework to minimize data movement between memory layers 33
34 Chapel Shared memory Distributed Memory Data Parallel Hello, world! config const numiters = ; Distributed memory const D = {1..numIters} dmapped Cyclic(startIdx=1); forall i in D do writeln( Hello, world!, from iteration, i, of, numiters); 34
35 Shasta Cray next generation HPC system for supporting both compute intensive and data intensive workloads Support for multiple infrastructures and software environments Adaptive supercomputing: support for different building blocks: Intel Xeon Intel Xeon Phi Intel Omni-Path network fabrics 35
36 Thank you
The Fusion of Supercomputing and Big Data. Peter Ungaro President & CEO
The Fusion of Supercomputing and Big Data Peter Ungaro President & CEO The Supercomputing Company Supercomputing Big Data Because some great things never change One other thing that hasn t changed. Cray
More informationOutline. High Performance Computing (HPC) Big Data meets HPC. Case Studies: Some facts about Big Data Technologies HPC and Big Data converging
Outline High Performance Computing (HPC) Towards exascale computing: a brief history Challenges in the exascale era Big Data meets HPC Some facts about Big Data Technologies HPC and Big Data converging
More informationVisualization and Data Analysis
Working Group Outbrief Visualization and Data Analysis James Ahrens, David Rogers, Becky Springmeyer Eric Brugger, Cyrus Harrison, Laura Monroe, Dino Pavlakos Scott Klasky, Kwan-Liu Ma, Hank Childs LLNL-PRES-481881
More informationParallel Computing. Benson Muite. benson.muite@ut.ee http://math.ut.ee/ benson. https://courses.cs.ut.ee/2014/paralleel/fall/main/homepage
Parallel Computing Benson Muite benson.muite@ut.ee http://math.ut.ee/ benson https://courses.cs.ut.ee/2014/paralleel/fall/main/homepage 3 November 2014 Hadoop, Review Hadoop Hadoop History Hadoop Framework
More informationData Analytics at NERSC. Joaquin Correa JoaquinCorrea@lbl.gov NERSC Data and Analytics Services
Data Analytics at NERSC Joaquin Correa JoaquinCorrea@lbl.gov NERSC Data and Analytics Services NERSC User Meeting August, 2015 Data analytics at NERSC Science Applications Climate, Cosmology, Kbase, Materials,
More informationSeeking Opportunities for Hardware Acceleration in Big Data Analytics
Seeking Opportunities for Hardware Acceleration in Big Data Analytics Paul Chow High-Performance Reconfigurable Computing Group Department of Electrical and Computer Engineering University of Toronto Who
More informationPilot-Streaming: Design Considerations for a Stream Processing Framework for High- Performance Computing
Pilot-Streaming: Design Considerations for a Stream Processing Framework for High- Performance Computing Andre Luckow, Peter M. Kasson, Shantenu Jha STREAMING 2016, 03/23/2016 RADICAL, Rutgers, http://radical.rutgers.edu
More informationCray s Storage History and Outlook Lustre+ Jason Goodman, Cray LUG 2015 - Denver
Cray s Storage History and Outlook Lustre+ Jason Goodman, Cray - Denver Agenda Cray History from Supercomputers to Lustre Where we are Today Cray Business OpenSFS Flashback to the Future SSDs, DVS, and
More informationEnabling High performance Big Data platform with RDMA
Enabling High performance Big Data platform with RDMA Tong Liu HPC Advisory Council Oct 7 th, 2014 Shortcomings of Hadoop Administration tooling Performance Reliability SQL support Backup and recovery
More informationData Centric Systems (DCS)
Data Centric Systems (DCS) Architecture and Solutions for High Performance Computing, Big Data and High Performance Analytics High Performance Computing with Data Centric Systems 1 Data Centric Systems
More informationBeyond Embarrassingly Parallel Big Data. William Gropp www.cs.illinois.edu/~wgropp
Beyond Embarrassingly Parallel Big Data William Gropp www.cs.illinois.edu/~wgropp Messages Big is big Data driven is an important area, but not all data driven problems are big data (despite current hype).
More informationHPC Trends and Directions in the Earth Sciences Per Nyberg nyberg@cray.com Sr. Director, Business Development
HPC Trends and Directions in the Earth Sciences Per Nyberg nyberg@cray.com Sr. Director, Business Development Cray Solutions for the Earth Sciences Why Cray? Cray solutions support: Range of modeling capabilities
More informationIntel HPC Distribution for Apache Hadoop* Software including Intel Enterprise Edition for Lustre* Software. SC13, November, 2013
Intel HPC Distribution for Apache Hadoop* Software including Intel Enterprise Edition for Lustre* Software SC13, November, 2013 Agenda Abstract Opportunity: HPC Adoption of Big Data Analytics on Apache
More informationDr. Raju Namburu Computational Sciences Campaign U.S. Army Research Laboratory. The Nation s Premier Laboratory for Land Forces UNCLASSIFIED
Dr. Raju Namburu Computational Sciences Campaign U.S. Army Research Laboratory 21 st Century Research Continuum Theory Theory embodied in computation Hypotheses tested through experiment SCIENTIFIC METHODS
More informationCray XC30 Hadoop Platform Jonathan (Bill) Sparks Howard Pritchard Martha Dumler
Cray XC30 Hadoop Platform Jonathan (Bill) Sparks Howard Pritchard Martha Dumler Safe Harbor Statement This presentation may contain forward-looking statements that are based on our current expectations.
More informationSo#ware Tools and Techniques for HPC, Clouds, and Server- Class SoCs Ron Brightwell
So#ware Tools and Techniques for HPC, Clouds, and Server- Class SoCs Ron Brightwell R&D Manager, Scalable System So#ware Department Sandia National Laboratories is a multi-program laboratory managed and
More informationA Brief Introduction to Apache Tez
A Brief Introduction to Apache Tez Introduction It is a fact that data is basically the new currency of the modern business world. Companies that effectively maximize the value of their data (extract value
More informationIn situ data analysis and I/O acceleration of FLASH astrophysics simulation on leadership-class system using GLEAN
In situ data analysis and I/O acceleration of FLASH astrophysics simulation on leadership-class system using GLEAN Venkatram Vishwanath 1, Mark Hereld 1, Michael E. Papka 1, Randy Hudson 2, G. Cal Jordan
More informationThe Fusion of Supercomputing and Big Data: The Role of Global Memory Architectures in Future Large Scale Data Analytics
HPC 2014 High Performance Computing FROM clouds and BIG DATA to EXASCALE AND BEYOND An International Advanced Workshop July 7 11, 2014, Cetraro, Italy Session III Emerging Systems and Solutions The Fusion
More informationHPC & Big Data THE TIME HAS COME FOR A SCALABLE FRAMEWORK
HPC & Big Data THE TIME HAS COME FOR A SCALABLE FRAMEWORK Barry Davis, General Manager, High Performance Fabrics Operation Data Center Group, Intel Corporation Legal Disclaimer Today s presentations contain
More informationHETEROGENEOUS HPC, ARCHITECTURE OPTIMIZATION, AND NVLINK
HETEROGENEOUS HPC, ARCHITECTURE OPTIMIZATION, AND NVLINK Steve Oberlin CTO, Accelerated Computing US to Build Two Flagship Supercomputers SUMMIT SIERRA Partnership for Science 100-300 PFLOPS Peak Performance
More informationSupercomputing on Windows. Microsoft (Thailand) Limited
Supercomputing on Windows Microsoft (Thailand) Limited W hat D efines S upercom puting A lso called High Performance Computing (HPC) Technical Computing Cutting edge problems in science, engineering and
More informationDeIC Watson Agreement - hvad betyder den for DeIC medlemmerne
DeIC Watson Agreement - hvad betyder den for DeIC medlemmerne Preben Jacobsen Solution Architect Nordic Lead, Software Defined Infrastructure Group IBM Danmark 2014 IBM Corporation Link: https://www.youtube.com/watch?v=_xcmh1lqb9i
More informationData-intensive HPC: opportunities and challenges. Patrick Valduriez
Data-intensive HPC: opportunities and challenges Patrick Valduriez Big Data Landscape Multi-$billion market! Big data = Hadoop = MapReduce? No one-size-fits-all solution: SQL, NoSQL, MapReduce, No standard,
More informationBig Data and Cloud Computing for GHRSST
Big Data and Cloud Computing for GHRSST Jean-Francois Piollé (jfpiolle@ifremer.fr) Frédéric Paul, Olivier Archer CERSAT / Institut Français de Recherche pour l Exploitation de la Mer Facing data deluge
More informationTHE EMC ISILON STORY. Big Data In The Enterprise. Copyright 2012 EMC Corporation. All rights reserved.
THE EMC ISILON STORY Big Data In The Enterprise 2012 1 Big Data In The Enterprise Isilon Overview Isilon Technology Summary 2 What is Big Data? 3 The Big Data Challenge File Shares 90 and Archives 80 Bioinformatics
More informationSupercomputing and Big Data: Where are the Real Boundaries and Opportunities for Synergy?
HPC2012 Workshop Cetraro, Italy Supercomputing and Big Data: Where are the Real Boundaries and Opportunities for Synergy? Bill Blake CTO Cray, Inc. The Big Data Challenge Supercomputing minimizes data
More informationManaging Complexity in Distributed Data Life Cycles Enhancing Scientific Discovery
Center for Information Services and High Performance Computing (ZIH) Managing Complexity in Distributed Data Life Cycles Enhancing Scientific Discovery Richard Grunzke*, Jens Krüger, Sandra Gesing, Sonja
More informationSC15 SYNOPSIS FOR FEDERAL GOVERNMENT
SC15 SYNOPSIS FOR FEDERAL GOVERNMENT SC15 Synopsis for Federal Government As a service to our clients, Engility offers a few notes from the Supercomputing Conference 2015 (SC15). Engility recently attended
More informationBig Data Analytics with Spark and Oscar BAO. Tamas Jambor, Lead Data Scientist at Massive Analytic
Big Data Analytics with Spark and Oscar BAO Tamas Jambor, Lead Data Scientist at Massive Analytic About me Building a scalable Machine Learning platform at MA Worked in Big Data and Data Science in the
More informationAutomating Big Data Benchmarking for Different Architectures with ALOJA
www.bsc.es Jan 2016 Automating Big Data Benchmarking for Different Architectures with ALOJA Nicolas Poggi, Postdoc Researcher Agenda 1. Intro on Hadoop performance 1. Current scenario and problematic 2.
More informationHPC data becomes Big Data. Peter Braam peter.braam@braamresearch.com
HPC data becomes Big Data Peter Braam peter.braam@braamresearch.com me 1983-2000 Academia Maths & Computer Science Entrepreneur with startups (5x) 4 startups sold Lustre emerged Held executive jobs with
More informationXeon+FPGA Platform for the Data Center
Xeon+FPGA Platform for the Data Center ISCA/CARL 2015 PK Gupta, Director of Cloud Platform Technology, DCG/CPG Overview Data Center and Workloads Xeon+FPGA Accelerator Platform Applications and Eco-system
More informationHPC ABDS: The Case for an Integrating Apache Big Data Stack
HPC ABDS: The Case for an Integrating Apache Big Data Stack with HPC 1st JTC 1 SGBD Meeting SDSC San Diego March 19 2014 Judy Qiu Shantenu Jha (Rutgers) Geoffrey Fox gcf@indiana.edu http://www.infomall.org
More informationEasier - Faster - Better
Highest reliability, availability and serviceability ClusterStor gets you productive fast with robust professional service offerings available as part of solution delivery, including quality controlled
More informationHPC Software Requirements to Support an HPC Cluster Supercomputer
HPC Software Requirements to Support an HPC Cluster Supercomputer Susan Kraus, Cray Cluster Solutions Software Product Manager Maria McLaughlin, Cray Cluster Solutions Product Marketing Cray Inc. WP-CCS-Software01-0417
More informationChallenges for Data Driven Systems
Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Quick History of Data Management 4000 B C Manual recording From tablets to papyrus to paper A. Payberah 2014 2
More informationBig Data - Infrastructure Considerations
April 2014, HAPPIEST MINDS TECHNOLOGIES Big Data - Infrastructure Considerations Author Anand Veeramani / Deepak Shivamurthy SHARING. MINDFUL. INTEGRITY. LEARNING. EXCELLENCE. SOCIAL RESPONSIBILITY. Copyright
More informationDeveloping Scalable Smart Grid Infrastructure to Enable Secure Transmission System Control
Developing Scalable Smart Grid Infrastructure to Enable Secure Transmission System Control EP/K006487/1 UK PI: Prof Gareth Taylor (BU) China PI: Prof Yong-Hua Song (THU) Consortium UK Members: Brunel University
More informationCisco UCS and Fusion- io take Big Data workloads to extreme performance in a small footprint: A case study with Oracle NoSQL database
Cisco UCS and Fusion- io take Big Data workloads to extreme performance in a small footprint: A case study with Oracle NoSQL database Built up on Cisco s big data common platform architecture (CPA), a
More informationHybrid Software Architectures for Big Data. Laurence.Hubert@hurence.com @hurence http://www.hurence.com
Hybrid Software Architectures for Big Data Laurence.Hubert@hurence.com @hurence http://www.hurence.com Headquarters : Grenoble Pure player Expert level consulting Training R&D Big Data X-data hot-line
More informationThe Ultimate in Scale-Out Storage for HPC and Big Data
Node Inventory Health and Active Filesystem Throughput Monitoring Asset Utilization and Capacity Statistics Manager brings to life powerful, intuitive, context-aware real-time monitoring and proactive
More informationPart I Courses Syllabus
Part I Courses Syllabus This document provides detailed information about the basic courses of the MHPC first part activities. The list of courses is the following 1.1 Scientific Programming Environment
More informationScaling Objectivity Database Performance with Panasas Scale-Out NAS Storage
White Paper Scaling Objectivity Database Performance with Panasas Scale-Out NAS Storage A Benchmark Report August 211 Background Objectivity/DB uses a powerful distributed processing architecture to manage
More informationBroadening Moab/TORQUE for Expanding User Needs
Broadening Moab/TORQUE for Expanding User Needs Gary D. Brown HPC Product Manager CUG 2016 1 2016 Adaptive Computing Enterprises, Inc. Agenda DataWarp Intel MIC KNL Viewpoint Web Portal User Portal Administrator
More informationData-Intensive Applications on HPC Using Hadoop, Spark and RADICAL-Cybertools
Data-Intensive Applications on HPC Using Hadoop, Spark and RADICAL-Cybertools Shantenu Jha, Andre Luckow, Ioannis Paraskevakos RADICAL, Rutgers, http://radical.rutgers.edu Agenda 1. Motivation and Background
More informationAccelerating Hadoop MapReduce Using an In-Memory Data Grid
Accelerating Hadoop MapReduce Using an In-Memory Data Grid By David L. Brinker and William L. Bain, ScaleOut Software, Inc. 2013 ScaleOut Software, Inc. 12/27/2012 H adoop has been widely embraced for
More informationLustre * Filesystem for Cloud and Hadoop *
OpenFabrics Software User Group Workshop Lustre * Filesystem for Cloud and Hadoop * Robert Read, Intel Lustre * for Cloud and Hadoop * Brief Lustre History and Overview Using Lustre with Hadoop Intel Cloud
More informationHadoop on the Gordon Data Intensive Cluster
Hadoop on the Gordon Data Intensive Cluster Amit Majumdar, Scientific Computing Applications Mahidhar Tatineni, HPC User Services San Diego Supercomputer Center University of California San Diego Dec 18,
More informationIntroducing EEMBC Cloud and Big Data Server Benchmarks
Introducing EEMBC Cloud and Big Data Server Benchmarks Quick Background: Industry-Standard Benchmarks for the Embedded Industry EEMBC formed in 1997 as non-profit consortium Defining and developing application-specific
More informationBuilding Platform as a Service for Scientific Applications
Building Platform as a Service for Scientific Applications Moustafa AbdelBaky moustafa@cac.rutgers.edu Rutgers Discovery Informa=cs Ins=tute (RDI 2 ) The NSF Cloud and Autonomic Compu=ng Center Department
More informationMaximizing Hadoop Performance and Storage Capacity with AltraHD TM
Maximizing Hadoop Performance and Storage Capacity with AltraHD TM Executive Summary The explosion of internet data, driven in large part by the growth of more and more powerful mobile devices, has created
More informationMaximize Performance and Scalability of RADIOSS* Structural Analysis Software on Intel Xeon Processor E7 v2 Family-Based Platforms
Maximize Performance and Scalability of RADIOSS* Structural Analysis Software on Family-Based Platforms Executive Summary Complex simulations of structural and systems performance, such as car crash simulations,
More informationRecent and Future Activities in HPC and Scientific Data Management Siegfried Benkner
Recent and Future Activities in HPC and Scientific Data Management Siegfried Benkner Research Group Scientific Computing Faculty of Computer Science University of Vienna AUSTRIA http://www.par.univie.ac.at
More informationPerformance Comparison of SQL based Big Data Analytics with Lustre and HDFS file systems
Performance Comparison of SQL based Big Data Analytics with Lustre and HDFS file systems Rekha Singhal and Gabriele Pacciucci * Other names and brands may be claimed as the property of others. Lustre File
More informationAppro Supercomputer Solutions Best Practices Appro 2012 Deployment Successes. Anthony Kenisky, VP of North America Sales
Appro Supercomputer Solutions Best Practices Appro 2012 Deployment Successes Anthony Kenisky, VP of North America Sales About Appro Over 20 Years of Experience 1991 2000 OEM Server Manufacturer 2001-2007
More informationKriterien für ein PetaFlop System
Kriterien für ein PetaFlop System Rainer Keller, HLRS :: :: :: Context: Organizational HLRS is one of the three national supercomputing centers in Germany. The national supercomputing centers are working
More informationHadoop MapReduce and Spark. Giorgio Pedrazzi, CINECA-SCAI School of Data Analytics and Visualisation Milan, 10/06/2015
Hadoop MapReduce and Spark Giorgio Pedrazzi, CINECA-SCAI School of Data Analytics and Visualisation Milan, 10/06/2015 Outline Hadoop Hadoop Import data on Hadoop Spark Spark features Scala MLlib MLlib
More informationBig Workflow: More than Just Intelligent Workload Management for Big Data
Big Workflow: More than Just Intelligent Workload Management for Big Data Michael Feldman White Paper February 2014 EXECUTIVE SUMMARY Big data applications represent a fast-growing category of high-value
More informationMaking Multicore Work and Measuring its Benefits. Markus Levy, president EEMBC and Multicore Association
Making Multicore Work and Measuring its Benefits Markus Levy, president EEMBC and Multicore Association Agenda Why Multicore? Standards and issues in the multicore community What is Multicore Association?
More informationHadoop MapReduce over Lustre* High Performance Data Division Omkar Kulkarni April 16, 2013
Hadoop MapReduce over Lustre* High Performance Data Division Omkar Kulkarni April 16, 2013 * Other names and brands may be claimed as the property of others. Agenda Hadoop Intro Why run Hadoop on Lustre?
More informationXSEDE Data Analytics Use Cases
XSEDE Data Analytics Use Cases 14th Jun 2013 Version 0.3 XSEDE Data Analytics Use Cases Page 1 Table of Contents A. Document History B. Document Scope C. Data Analytics Use Cases XSEDE Data Analytics Use
More informationBig Data, Biggestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgestgest
Worlds Colliding: Why Big Data Changes How To Think About Enterprise Storage Addison Snell addison@intersect360.com Overview, Mostly in Order Okay, who invited the HPC guy? The hazards of forecasting Big
More informationUnlocking the True Value of Hadoop with Open Data Science
Unlocking the True Value of Hadoop with Open Data Science Kristopher Overholt Solution Architect Big Data Tech 2016 MinneAnalytics June 7, 2016 Overview Overview of Open Data Science Python and the Big
More informationWrite a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical
Identify a problem Review approaches to the problem Propose a novel approach to the problem Define, design, prototype an implementation to evaluate your approach Could be a real system, simulation and/or
More informationPetascale Software Challenges. Piyush Chaudhary piyushc@us.ibm.com High Performance Computing
Petascale Software Challenges Piyush Chaudhary piyushc@us.ibm.com High Performance Computing Fundamental Observations Applications are struggling to realize growth in sustained performance at scale Reasons
More informationSun Constellation System: The Open Petascale Computing Architecture
CAS2K7 13 September, 2007 Sun Constellation System: The Open Petascale Computing Architecture John Fragalla Senior HPC Technical Specialist Global Systems Practice Sun Microsystems, Inc. 25 Years of Technical
More informationThe PHI solution. Fujitsu Industry Ready Intel XEON-PHI based solution. SC2013 - Denver
1 The PHI solution Fujitsu Industry Ready Intel XEON-PHI based solution SC2013 - Denver Industrial Application Challenges Most of existing scientific and technical applications Are written for legacy execution
More informationMAKING THE BUSINESS CASE
MAKING THE BUSINESS CASE LUSTRE FILE SYSTEMS ARE POISED TO PENETRATE COMMERCIAL MARKETS table of contents + Considerations in Building the.... 1... 3.... 4 A TechTarget White Paper by Long the de facto
More informationLambda Architecture. Near Real-Time Big Data Analytics Using Hadoop. January 2015. Email: bdg@qburst.com Website: www.qburst.com
Lambda Architecture Near Real-Time Big Data Analytics Using Hadoop January 2015 Contents Overview... 3 Lambda Architecture: A Quick Introduction... 4 Batch Layer... 4 Serving Layer... 4 Speed Layer...
More informationCluster, Grid, Cloud Concepts
Cluster, Grid, Cloud Concepts Kalaiselvan.K Contents Section 1: Cluster Section 2: Grid Section 3: Cloud Cluster An Overview Need for a Cluster Cluster categorizations A computer cluster is a group of
More informationOracle Big Data SQL Technical Update
Oracle Big Data SQL Technical Update Jean-Pierre Dijcks Oracle Redwood City, CA, USA Keywords: Big Data, Hadoop, NoSQL Databases, Relational Databases, SQL, Security, Performance Introduction This technical
More informationBSC vision on Big Data and extreme scale computing
BSC vision on Big Data and extreme scale computing Jesus Labarta, Eduard Ayguade,, Fabrizio Gagliardi, Rosa M. Badia, Toni Cortes, Jordi Torres, Adrian Cristal, Osman Unsal, David Carrera, Yolanda Becerra,
More informationBig Fast Data Hadoop acceleration with Flash. June 2013
Big Fast Data Hadoop acceleration with Flash June 2013 Agenda The Big Data Problem What is Hadoop Hadoop and Flash The Nytro Solution Test Results The Big Data Problem Big Data Output Facebook Traditional
More informationScaling Across the Supercomputer Performance Spectrum
Scaling Across the Supercomputer Performance Spectrum Cray s XC40 system leverages the combined advantages of next-generation Aries interconnect and Dragonfly network topology, Intel Xeon processors, integrated
More informationANALYTICS IN BIG DATA ERA
ANALYTICS IN BIG DATA ERA ANALYTICS TECHNOLOGY AND ARCHITECTURE TO MANAGE VELOCITY AND VARIETY, DISCOVER RELATIONSHIPS AND CLASSIFY HUGE AMOUNT OF DATA MAURIZIO SALUSTI SAS Copyr i g ht 2012, SAS Ins titut
More informationBig Data for Big Value @ Intel
Big Data for Big Value @ Intel Moty Fania, PE Big data Analytics Assaf Araki, Sr. Arch. Big data Analytics Advanced Analytics team @ Intel IT Corporate ownership of advanced analytics Team charter Solve
More informationIBM Power Systems This is Power on a Smarter Planet
IBM Power Systems This is Power on a Smarter Planet Red Hat Enterprise Linux for IBM Power Systems! Filipe Miranda Global Lead for Linux on IBM System z and Power Systems!, #powerlinux, #bigdata, #IBMWatson,
More informationCray: Enabling Real-Time Discovery in Big Data
Cray: Enabling Real-Time Discovery in Big Data Discovery is the process of gaining valuable insights into the world around us by recognizing previously unknown relationships between occurrences, objects
More informationYALES2 porting on the Xeon- Phi Early results
YALES2 porting on the Xeon- Phi Early results Othman Bouizi Ghislain Lartigue Innovation and Pathfinding Architecture Group in Europe, Exascale Lab. Paris CRIHAN - Demi-journée calcul intensif, 16 juin
More informationSpark: Cluster Computing with Working Sets
Spark: Cluster Computing with Working Sets Outline Why? Mesos Resilient Distributed Dataset Spark & Scala Examples Uses Why? MapReduce deficiencies: Standard Dataflows are Acyclic Prevents Iterative Jobs
More informationBuilding an energy dashboard. Energy measurement and visualization in current HPC systems
Building an energy dashboard Energy measurement and visualization in current HPC systems Thomas Geenen 1/58 thomas.geenen@surfsara.nl SURFsara The Dutch national HPC center 2H 2014 > 1PFlop GPGPU accelerators
More informationBig Data Research in Berlin BBDC and Apache Flink
Big Data Research in Berlin BBDC and Apache Flink Tilmann Rabl rabl@tu-berlin.de dima.tu-berlin.de bbdc.berlin 1 2013 Berlin Big Data Center All Rights Reserved DIMA 2015 Agenda About Data Management,
More informationWrangler: A New Generation of Data-intensive Supercomputing. Christopher Jordan, Siva Kulasekaran, Niall Gaffney
Wrangler: A New Generation of Data-intensive Supercomputing Christopher Jordan, Siva Kulasekaran, Niall Gaffney Project Partners Academic partners: TACC Primary system design, deployment, and operations
More informationThe virtualization of SAP environments to accommodate standardization and easier management is gaining momentum in data centers.
White Paper Virtualized SAP: Optimize Performance with Cisco Data Center Virtual Machine Fabric Extender and Red Hat Enterprise Linux and Kernel-Based Virtual Machine What You Will Learn The virtualization
More informationCluster Scalability of ANSYS FLUENT 12 for a Large Aerodynamics Case on the Darwin Supercomputer
Cluster Scalability of ANSYS FLUENT 12 for a Large Aerodynamics Case on the Darwin Supercomputer Stan Posey, MSc and Bill Loewe, PhD Panasas Inc., Fremont, CA, USA Paul Calleja, PhD University of Cambridge,
More informationHigh Performance Computing (HPC)
High Performance Computing (HPC) High Performance Computing (HPC) White Paper Attn: Name, Title Phone: xxx.xxx.xxxx Fax: xxx.xxx.xxxx 1.0 OVERVIEW When heterogeneous enterprise environments are involved,
More informationApache Flink Next-gen data analysis. Kostas Tzoumas ktzoumas@apache.org @kostas_tzoumas
Apache Flink Next-gen data analysis Kostas Tzoumas ktzoumas@apache.org @kostas_tzoumas What is Flink Project undergoing incubation in the Apache Software Foundation Originating from the Stratosphere research
More informationReference Architecture, Requirements, Gaps, Roles
Reference Architecture, Requirements, Gaps, Roles The contents of this document are an excerpt from the brainstorming document M0014. The purpose is to show how a detailed Big Data Reference Architecture
More informationSoftware-defined Storage Architecture for Analytics Computing
Software-defined Storage Architecture for Analytics Computing Arati Joshi Performance Engineering Colin Eldridge File System Engineering Carlos Carrero Product Management June 2015 Reference Architecture
More informationBig Data Research in the AMPLab: BDAS and Beyond
Big Data Research in the AMPLab: BDAS and Beyond Michael Franklin UC Berkeley 1 st Spark Summit December 2, 2013 UC BERKELEY AMPLab: Collaborative Big Data Research Launched: January 2011, 6 year planned
More informationOverview of HPC Resources at Vanderbilt
Overview of HPC Resources at Vanderbilt Will French Senior Application Developer and Research Computing Liaison Advanced Computing Center for Research and Education June 10, 2015 2 Computing Resources
More informationAppendix 1 ExaRD Detailed Technical Descriptions
Appendix 1 ExaRD Detailed Technical Descriptions Contents 1 Application Foundations... 3 1.1 Co- Design... 3 1.2 Applied Mathematics... 5 1.3 Data Analytics and Visualization... 8 2 User Experiences...
More informationCloud Computing for Research Roger Barga Cloud Computing Futures, Microsoft Research
Cloud Computing for Research Roger Barga Cloud Computing Futures, Microsoft Research Trends: Data on an Exponential Scale Scientific data doubles every year Combination of inexpensive sensors + exponentially
More informationHur hanterar vi utmaningar inom området - Big Data. Jan Östling Enterprise Technologies Intel Corporation, NER
Hur hanterar vi utmaningar inom området - Big Data Jan Östling Enterprise Technologies Intel Corporation, NER Legal Disclaimers All products, computer systems, dates, and figures specified are preliminary
More informationData Lake In Action: Real-time, Closed Looped Analytics On Hadoop
1 Data Lake In Action: Real-time, Closed Looped Analytics On Hadoop 2 Pivotal s Full Approach It s More Than Just Hadoop Pivotal Data Labs 3 Why Pivotal Exists First Movers Solve the Big Data Utility Gap
More informationHPC and Big Data. EPCC The University of Edinburgh. Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk
HPC and Big Data EPCC The University of Edinburgh Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk EPCC Facilities Technology Transfer European Projects HPC Research Visitor Programmes Training
More informationLS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance
11 th International LS-DYNA Users Conference Session # LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance Gilad Shainer 1, Tong Liu 2, Jeff Layton 3, Onur Celebioglu
More informationLinux Performance Optimizations for Big Data Environments
Linux Performance Optimizations for Big Data Environments Dominique A. Heger Ph.D. DHTechnologies (Performance, Capacity, Scalability) www.dhtusa.com Data Nubes (Big Data, Hadoop, ML) www.datanubes.com
More informationEinsatzfelder von IBM PureData Systems und Ihre Vorteile.
Einsatzfelder von IBM PureData Systems und Ihre Vorteile demirkaya@de.ibm.com Agenda Information technology challenges PureSystems and PureData introduction PureData for Transactions PureData for Analytics
More information