Big Data and Scientific Discovery
|
|
|
- Ronald Todd
- 8 years ago
- Views:
Transcription
1 Big Data and Scientific Discovery Bill Harrod Office of Science February 26, 2014!
2 Big Data and Scien*fic Discovery Next genera*on scien*fic breakthroughs require: Major new advances in compu*ng technology: energy- efficient hardware, algorithms, applica6ons, system so7ware Dealing with big data : advances in data management, data analy6cs, visualiza6on and machine learning algorithms Ubiquitous data explosion Experimental, observa6onal, sensor networks, and simula6on Compu*ng / data throughput challenges Can no longer scale up from or modify exis6ng solu6ons Problem compounded by need for interna6onal collabora6on/coordina6on Complicated workflows 1!
3 Extreme Scale Science Next Genera*on of Scien*fic Discovery Genomics Data Volume increases to 10 PB in FY21 High Energy Physics (Large Hadron Collider) 15 PB of data/year Light Sources Approximately 300 TB/day Climate Data expected to be 100 EB Data explosion is driven by exponen*al technology advances Data sources Scien6fic Instruments Scien6fic Compu6ng Facili6es Simula6on Results Observa6onal data Big Data and Big Compute Analyzing Big Data requires processing (e.g., search, transform, analyze, ) Extreme scale compu6ng will enable 6mely and more complex processing of increasingly large data sets Very few large scale applica*ons of prac*cal importance are NOT data intensive. Alok Choudhary, IESP, Kobe, Japan, April 2012 February 26, 2014! 2!
4 DOE Office of Science Computa*onal Facili*es Titan System #1! Specifications: Peak performance of 27.1 November Petaflops! GPU CPU! 18,688 Hybrid Compute Nodes with:! 16-Core AMD Opteron CPU! NVIDIA Tesla K20x GPU! GB memory! Mira System Specifications:! 200 Cabinets; 710 TB total system Peak performance of 10 Petaflops! memory; 8.9 MW peak power! 49,152 Compute Nodes each with:! 16-Core Power PC A2 CPU with 64 Hardware Threads and 16 Quad FPUs! 16 GB memory! 56 Cabinets; 786 TB total system memory; 4.8 MW peak power! NERSC System Specifications:! Hopper XT5 (2010)! 1.3PF, 212TB, 2.9 MW peak power! Edison XC30! Based on DARPA/DOE HPCS system! 2.4PF, 333TB, 2.1 MW peak power! ASCR Briefing 09/03/2013! 3!
5 Energy Science Network (ESnet) 100 Gigabit Upgrade: Project Complete Network Specifications:! Connects 40 DOE sites to internet! Growing 2x commercial nets 50% of traffic is from big data! First 100G cross USA! Network can grow to 88 independent 100G channels! Deploying 100G production connections at ANL, BNL, FNAL, LBNL, LLNL, NERSC, ORNL! February 26, 2014! 4!
6 One Possible View of the Future Individual University and Lab PIs! National and Int l collabs! Research + Industry! Industry! Scientific Discovery Engines! Common Services! ESnet! Exascale Simulation Facilities! Massive Throughput Simulations! Data Serving and Archiving Facilities! Analytics & Visualization Systems! ESnet! Local Nodes! Local Nodes! Local Nodes! Local Nodes! Local Nodes! Light sources, etc. (BES)! Colliders (HEP/NP)! Sequencers (BER)! Cosmology (HEP)! Omics data (All)! February 26, 2014! 5
7 Scien*fic Discovery Environments Current environment not op*mal for: Geographically distributed data, users, and facili6es Dynamic, interac6ve workflow Computa6onal steering and real- 6me in- situ analysis Future need for shared, real- *me visualiza*on, comparison of simula*ons with expt l results, and control of ac*ve processes Interna*onal collabora*on can push the boundaries of scien*fic discovery by crea*ng unified interna*onal teams from distributed users and facili*es Collabora*on is inherently a big data issue February 26, 2014! 6!
8 Uncertainty Threatens Future of Compu*ng The world has changed technology is changing at a drama*c rate Dennard scaling has ended End of Moore's Law looming The IT marketplace is also changing drama*cally PC sales have fla]ened Handhelds dominate growth, H/W and S/W HPC vendor uncertainty Need to drive innova*ons at all levels of technology Processors & Memory (Node) System Designs System So7ware Algorithms Data volume and variety explosion Data management Analy6cs / visualiza6on / machine learning February 26, 2014! 7!
9 Exascale Compu*ng The Vision Exascale compu*ng Achieve order opera6ons per second and order bytes of storage Address the next genera6on of scien6fic, engineering, and large- data workflows Enable extreme scale compu6ng: 1,000X capabili6es of today's computers with a similar size and power footprint Establish a new trajectory of progress towards a broad spectrum of compu6ng capabili6es over the succeeding decade Produc*ve system Usable by a wide variety of scien6sts and engineers Easier to develop so7ware & management of the system Based on marketable technology Not a one off system Scalable, sustainable technology, exploi6ng economies of scale and trickle- bounce effect Deployed in early 2020s February 26, 2014! 8!
10 Advanced Scien+fic Compu+ng Advisory Commi6ee Top Ten Technical Approaches for Exascale 1. Energy efficiency: Crea6ng more energy efficient circuit, power, and cooling technologies. 2. Interconnect technology: Increasing the performance and energy efficiency of data movement. 3. Memory technology: Integra6ng advanced memory technologies to improve both capacity and bandwidth. 4. Scalable System Sodware: Developing scalable system so7ware that is power and resilience aware. 5. Programming systems: Inven6ng new programming environments that express massive parallelism, data locality, and resilience 6. Data management: Crea6ng data management so7ware that can handle the volume, velocity and diversity of data that is an6cipated. 7. Exascale algorithms: Reformula6ng science problems and refactoring their solu6on algorithms for exascale systems. 8. Algorithms for discovery, design, and decision: Facilita6ng mathema6cal op6miza6on and uncertainty quan6fica6on for exascale discovery, design, and decision making. 9. Resilience and correctness: Ensuring correct scien6fic computa6on in face of faults, reproducibility, and algorithm verifica6on challenges. 10. Scien*fic produc*vity: Increasing the produc6vity of computa6onal scien6sts with new so7ware engineering tools and environments. hjp://science.energy.gov/~/media/ascr/ascac/pdf/reports/2013/report.pdf February 26, 2014! 9!
11 Scien*fic Data Challenges Workflows for computa*onal science must drive fundamental changes in computer architecture for exascale systems Breaking with the past: tradi+onal scien+fic workflow - simulate or experiment, saving the data to disk for later analysis Worsening I/O bojleneck and energy cost of data movement combine to make it impossible to save all of the data to disk in situ data analysis, occurring on the supercomputer while the simula*on is running. February 26, 2014! 10!
12 Data Management, Analysis and Visualiza*on Top 10 Technical Approaches 1. Data structures and traversal algorithms that minimize data movement 2. Methods for data reduc*on/triage that support valida6on of results and data repurposing 3. Maintaining the ability to do exploratory analysis to discover the unexpected despite severe data reduc6on 4. Knowledge representa*on and machine reasoning to capture and use data provenance 5. Coordina6on of resource access among running simula6ons and data management, analysis and visualiza6on technologies that run in situ 6. Methods of in situ data analysis that minimize reliance on a priori knowledge 7. Data analysis algorithms for high- velocity, high- volume mul*- sensor, mul*- resolu*on data 8. Methods for compara*ve and/or integrated analysis of simula6on and experimental/ observa6onal data 9. Design of sharable in situ scien*fic workflows to support data management, processing, analysis and visualiza6on 10. Maintaining data integrity in the face of error- prone systems 11. Methods of visual analysis for data sets at scale and metrics for valida6ng them 12. Improved abstrac6ons for data storage that move beyond the concept of files to more richly represent the scien*fic seman*cs of experiments, simula*ons, and data points Lucy Nowell, DOE/ASCR February 26, 2014! 11!
13 Use of Co- Design in DOE Exascale Strategy Approach: Use scien*fic workflow requirements to guide architecture and system sodware and use technology capabili*es to design algorithms and sodware Three co- design centers focus on specific applica6on domains: ExaCT: Combus6on simula6on (uniform and adap6ve mesh) ExMatEx: Materials (mul6ple codes) CESAR: Nuclear engineering (structures, fluids, transport) Create proxy apps Scaled down versions of full code Selects parts/pa]erns from code to drive programming/ architecture
14 Exascale Compu*ng Ini*a*ve Proposed Timeline Platform Acquisitions Application Development! Future Computer Systems: Pathway Towards Exascale! Science, Engineering and Defense Applications! Research & Development! Exascale Co-Design: Driving the design of Exascale HW and SW! Design Forward! System Design Phase! Prototype Build Phase! Fast Forward! Path Forward Phase! Software Technology: Programming Environment, Resiliency, OS & Runtimes! Extreme Scale Research Programs (SC/ASCR & NNSA/ASC): Fundamental Technology! P0! P1! P2! FY! 2012! 2013! 2014! 2015! 2016! 2017! 2018! 2019! 2020! 2021! 2022! 2023! 2024! Node Prototype! Petascale Prototype! P0! P1! P2! Exascale Prototype!
15 Summary Four Primary Research Drivers Ensuring a con*nued, sustainable, and strong technology base that is energy efficient and high performance Computer technology is a cri6cal element in our daily life, economic vitality, scien6fic advances and na6onal security Increasing the produc*vity of computa*onal scien*sts Making high- end compu6ng easier will lower the barriers to broad adop6on of HPC technologies by industrial and business user Achieving the full poten*al of the scien*fic discovery environment Today's capabili6es are the result of a piecemeal research approach Need infrastructure for interna6onal collabora6on Enable scien*fic workflow ecosystems that supports sharing, collabora*on, and reuse of workflows February 26, 2014! 14!
16 BACKUP SLIDES February 26, 2014! 15!
17 Exascale Compu*ng We Need to Reinvent Compu*ng! Tradi*onal path of 2x performance improvement every 18 months has ended For decades, Moore's Law plus Dennard scaling provided more, faster transistors in each new process technology This is no longer true we have hit a power wall! The result is unacceptable power requirements for increased performance We cannot procure an exascale system based on today's or projected future commodity technology Exis6ng HPC solu6ons cannot be usefully scaled up to exascale Energy consump6on would be prohibi6ve (~300MW) Exascale will require partnering with U.S. compu*ng industry to chart the future Industry at a crossroads and is open to new paths Time is right to push energy efficiency into the marketplace! February 26, 2014! 16!
18 Exascale Challenges Four primary challenges must be overcome Parallelism / concurrency Reliability / resiliency Energy efficiency Memory / Storage Produc*vity issues Managing system complexity Portability Generality System design issues Scalability Efficiency Time to solu6on Dependability (security and reliability) must be integrated at all levels of the design Most provide op6mal performance for work flow February 26, 2014! 17!
19 Exascale Compu*ng Ini*a*ve (ECI) Target System Characteris*cs 20 pj per average opera*on ~ x 40 improvement over today's efficiency Billion- way concurrency Ecosystem to support new applica*on development and collabora*ve work, enable transparent portability, accommodate legacy applica*ons High reliability and resilience through self- diagnos*cs and self- healing Programming environments (high- level languages, tools, ) to increase scien*fic produc*vity February 26, 2014! 18!
20 Extreme Scale Science Work Flow Observa*onal Data! Esnet (network)! DOE User Computa*onal Facility! Experimental Data! DOE User Experimental Facility! Local compute, storage and network resources! Scien*fic Discovery! February 26, 2014! 19!
21 Scien*fic Data Issues For purely in situ analysis, need to know a priori what analysis is needed Consider the value of automa6ng exploratory analysis or hypothesis genera6on. How to capture provenance and make the informa*on usable and useful for valida*on of results. Need resilient data management, analysis and visualiza*on sodware How to help scien*sts understand tremendous volumes of data. February 26, 2014! 20!
NERSC Data Efforts Update Prabhat Data and Analytics Group Lead February 23, 2015
NERSC Data Efforts Update Prabhat Data and Analytics Group Lead February 23, 2015-1 - A little bit about myself Computer Scien.st Brown, IIT Delhi Real- 3me Graphics, Virtual Reality, HCI Computa3onal
OS/Run'me and Execu'on Time Produc'vity
OS/Run'me and Execu'on Time Produc'vity Ron Brightwell, Technical Manager Scalable System SoAware Department Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation,
Data Centric Systems (DCS)
Data Centric Systems (DCS) Architecture and Solutions for High Performance Computing, Big Data and High Performance Analytics High Performance Computing with Data Centric Systems 1 Data Centric Systems
Processing/ Processing expensive/ Processing free/ Memory: memory free memory expensive
DOE Exascale Initiative Dimitri Kusnezov, Senior Advisor to the Secretary, US DOE Steve Binkley, Senior Advisor, Office of Science, US DOE Bill Harrod, Office of Science/ASCR Bob Meisner, Defense Programs/ASC
Exascale Computing Project (ECP) Update
Exascale Computing Project (ECP) Update Presented to ASCAC Paul Messina Project Director Stephen Lee Deputy Director Washington, D.C. April 4, 2016 ECP is in the process of making the transition from an
Behind the scene III Cloud computing
Behind the scene III Cloud computing Athens, 15.11.2014 M. Dolenc / R. Klinc Why we do it? Engineering in the cloud is a combina3on of cloud based services and rich interac3ve applica3ons allowing engineers
Data Management in the Cloud: Limitations and Opportunities. Annies Ductan
Data Management in the Cloud: Limitations and Opportunities Annies Ductan Discussion Outline: Introduc)on Overview Vision of Cloud Compu8ng Managing Data in The Cloud Cloud Characteris8cs Data Management
BENCHMARKING V ISUALIZATION TOOL
Copyright 2014 Splunk Inc. BENCHMARKING V ISUALIZATION TOOL J. Green Computer Scien
Big Data. The Big Picture. Our flexible and efficient Big Data solu9ons open the door to new opportuni9es and new business areas
Big Data The Big Picture Our flexible and efficient Big Data solu9ons open the door to new opportuni9es and new business areas What is Big Data? Big Data gets its name because that s what it is data that
Components of Technology Suppor4ng Data Intensive Research
Components of Technology Suppor4ng Data Intensive Research Ron Hutchins Associate Vice Provost for Research and Technology and CTO Georgia Ins4tute of Technology 24 January, 2012 NSF Dear Colleague LeKer:
Big Data and Clouds: Challenges and Opportuni5es
Big Data and Clouds: Challenges and Opportuni5es NIST January 15 2013 Geoffrey Fox gcf@indiana.edu h"p://www.infomall.org h"p://www.futuregrid.org School of Informa;cs and Compu;ng Digital Science Center
Making Sense of Big Data. Dr. Thomas E. Potok Computa2onal Data Analy2cs Group Leader Oak Ridge Na2onal Laboratory potokte@ornl.
Making Sense of Big Data Dr. Thomas E. Potok Computa2onal Data Analy2cs Group Leader Oak Ridge Na2onal Laboratory potokte@ornl.gov 865-574- 0834 ORNL s Big Data Legacy Science National Security Energy
B2B Offerings. Helping businesses op2mize. Infolob s amazing b2b offerings helps your company achieve maximum produc2vity
B2B Offerings Helping businesses op2mize Infolob s amazing b2b offerings helps your company achieve maximum produc2vity What is B2B? B2B is shorthand for the sales prac4ce called business- to- business
An Open Dynamic Big Data Driven Applica3on System Toolkit
An Open Dynamic Big Data Driven Applica3on System Toolkit Craig C. Douglas University of Wyoming and KAUST This research is supported in part by the Na3onal Science Founda3on and King Abdullah University
Data Center Evolu.on and the Cloud. Paul A. Strassmann George Mason University November 5, 2008, 7:20 to 10:00 PM
Data Center Evolu.on and the Cloud Paul A. Strassmann George Mason University November 5, 2008, 7:20 to 10:00 PM 1 Hardware Evolu.on 2 Where is hardware going? x86 con(nues to move upstream Massive compute
Engagement Strategies for Emerging Big Data Collaborations
Engagement Strategies for Emerging Big Data Collaborations Lauren Rotman, lauren@es.net ESnet Science Engagement Group Lead Lawrence Berkeley National Laboratory APAN 39 th Conference Global Collaborations
Data Analytics at NERSC. Joaquin Correa JoaquinCorrea@lbl.gov NERSC Data and Analytics Services
Data Analytics at NERSC Joaquin Correa JoaquinCorrea@lbl.gov NERSC Data and Analytics Services NERSC User Meeting August, 2015 Data analytics at NERSC Science Applications Climate, Cosmology, Kbase, Materials,
The Real Score of Cloud
The Real Score of Cloud Mayur Sahni Sr. Research Manger IDC Asia/Pacific msahni@idc.com @mayursahni Digital Transformation Changing Role of IT Innova&on Informa&on Business agility Changing role of the
Hank Childs, University of Oregon
Exascale Analysis & Visualization: Get Ready For a Whole New World Sept. 16, 2015 Hank Childs, University of Oregon Before I forget VisIt: visualization and analysis for very big data DOE Workshop for
Cloud Compu)ng in Educa)on and Research
Cloud Compu)ng in Educa)on and Research Dr. Wajdi Loua) Sfax University, Tunisia ESPRIT - December 2014 04/12/14 1 Outline Challenges in Educa)on and Research SaaS, PaaS and IaaS for Educa)on and Research
SQream Technologies Ltd - Confiden7al
SQream Technologies Ltd - Confiden7al 1 Ge#ng Big Data Done On a GPU- Based Database Ori Netzer VP Product 26- Mar- 14 Analy7cs Performance - 3 TB, 18 Billion records SQream Database 400x More Cost Efficient!
Workshop on Exascale Data Management, Analysis, and Visualiza=on Houston TX 2/22/2011 Scott A. Klasky
Workshop on Exascale Data Management, Analysis, and Visualiza=on Houston TX 2/22/2011 Scott A. Klasky klasky@ornl.gov ORNL: Q. Liu, J. Logan, N. Podhorszki, R. Tchoua Georgia Tech: H. Abbasi, G. Eisehnhauer,
Future Trends in Advanced Scientific Computing for Open Science
Future Trends in Advanced Scientific Computing for Open Science Presented to the High-Energy Physics Advisory Panel by Steve Binkley DOE Office of Advanced Scientific Computing Research Steve.Binkley@science.doe.gov
Panasas High Performance Storage Powers the First Petaflop Supercomputer at Los Alamos National Laboratory
Customer Success Story Los Alamos National Laboratory Panasas High Performance Storage Powers the First Petaflop Supercomputer at Los Alamos National Laboratory June 2010 Highlights First Petaflop Supercomputer
Visualization and Data Analysis
Working Group Outbrief Visualization and Data Analysis James Ahrens, David Rogers, Becky Springmeyer Eric Brugger, Cyrus Harrison, Laura Monroe, Dino Pavlakos Scott Klasky, Kwan-Liu Ma, Hank Childs LLNL-PRES-481881
Research at the Department of Computer Science and Software Engineering. Professor Yong Yue BEng, PhD, CEng, FIET, FIMechE 17 October 2014
Research at the Department of Computer Science and Software Engineering Professor Yong Yue BEng, PhD, CEng, FIET, FIMechE 17 October 2014 Research Areas Ar%ficial intelligence Robo%cs Data mining Image
Make the Most of Big Data to Drive Innovation Through Reseach
White Paper Make the Most of Big Data to Drive Innovation Through Reseach Bob Burwell, NetApp November 2012 WP-7172 Abstract Monumental data growth is a fact of life in research universities. The ability
Data Requirements from NERSC Requirements Reviews
Data Requirements from NERSC Requirements Reviews Richard Gerber and Katherine Yelick Lawrence Berkeley National Laboratory Summary Department of Energy Scientists represented by the NERSC user community
Overview on Modern Accelerators and Programming Paradigms Ivan Giro7o igiro7o@ictp.it
Overview on Modern Accelerators and Programming Paradigms Ivan Giro7o igiro7o@ictp.it Informa(on & Communica(on Technology Sec(on (ICTS) Interna(onal Centre for Theore(cal Physics (ICTP) Mul(ple Socket
Appendix 1 ExaRD Detailed Technical Descriptions
Appendix 1 ExaRD Detailed Technical Descriptions Contents 1 Application Foundations... 3 1.1 Co- Design... 3 1.2 Applied Mathematics... 5 1.3 Data Analytics and Visualization... 8 2 User Experiences...
Portable, Scalable, and High-Performance I/O Forwarding on Massively Parallel Systems. Jason Cope copej@mcs.anl.gov
Portable, Scalable, and High-Performance I/O Forwarding on Massively Parallel Systems Jason Cope copej@mcs.anl.gov Computation and I/O Performance Imbalance Leadership class computa:onal scale: >100,000
Big process for big data
Big process for big data Process automa9on for data- driven science Ian Foster Computa9on Ins9tute Argonne Na9onal Laboratory & The University of Chicago Talk at Astroinforma9cs 2012, Redmond, September
Data Center 2020. DC planning for the next 5 10 years. Copyright 2004-2013 Experture and Robert Frances Group, all rights reserved
DC planning for the next 5 10 years Topics to be Discussed Introduc=on Indirect Drivers Technology Direct Drivers Data Center DC Management DC Opera=ons s and Disaster Recovery 2 Introduc=on The future
Experiments on cost/power and failure aware scheduling for clouds and grids
Experiments on cost/power and failure aware scheduling for clouds and grids Jorge G. Barbosa, Al0no M. Sampaio, Hamid Harabnejad Universidade do Porto, Faculdade de Engenharia, LIACC Porto, Portugal, jbarbosa@fe.up.pt
Project Por)olio Management
Project Por)olio Management Important markers for IT intensive businesses Rest assured with Infolob s project management methodologies What is Project Por)olio Management? Project Por)olio Management (PPM)
Mission. To provide higher technological educa5on with quality, preparing. competent professionals, with sound founda5ons in science, technology
Mission To provide higher technological educa5on with quality, preparing competent professionals, with sound founda5ons in science, technology and innova5on, commi
Achieving Performance Isolation with Lightweight Co-Kernels
Achieving Performance Isolation with Lightweight Co-Kernels Jiannan Ouyang, Brian Kocoloski, John Lange The Prognostic Lab @ University of Pittsburgh Kevin Pedretti Sandia National Laboratories HPDC 2015
Mobility in the Modern Factory. Discussion of Mobile Adop7on for the Factories of the Future
Mobility in the Modern Factory Discussion of Mobile Adop7on for the Factories of the Future Talking Points History Lesson The Reasons for Going Mobile Mobile Infrastructure Mobile Device Security BYOD
Sun Constellation System: The Open Petascale Computing Architecture
CAS2K7 13 September, 2007 Sun Constellation System: The Open Petascale Computing Architecture John Fragalla Senior HPC Technical Specialist Global Systems Practice Sun Microsystems, Inc. 25 Years of Technical
Informa*on Management
Informa*on Management Deepak Mohan SVP, Informa3on Management Group 1 Symantec Informa*on Management Strategy Protect Completely Dedupe Everywhere Delete Confidently Discover Efficiently Backup, archive
Presenta<on to EMA GCP IWG. Cloud Services - A Framework for Adop<on in the Regulated Life Sciences Industry. Agenda item 03.1.1
Agenda item 03.1.1 Formed in 2004 >6000 members worldwide Not- for- profit organiza
Technology Big Data Solutions for Aeronautics : value, issues and solution. Business Models. Usage
Technology Big Data Solutions for Aeronautics : value, issues and solution Business Models Usage Content 1. Big Data services for aerospace 2. Altran approach: VueForge TM 3. VueForge TM for Automotive
So#ware Tools and Techniques for HPC, Clouds, and Server- Class SoCs Ron Brightwell
So#ware Tools and Techniques for HPC, Clouds, and Server- Class SoCs Ron Brightwell R&D Manager, Scalable System So#ware Department Sandia National Laboratories is a multi-program laboratory managed and
Big Data Research at DKRZ
Big Data Research at DKRZ Michael Lautenschlager and Colleagues from DKRZ and Scien:fic Compu:ng Research Group Symposium Big Data in Science Karlsruhe October 7th, 2014 Big Data in Climate Research Big
High Performance Compu2ng Facility
High Performance Compu2ng Facility Center for Health Informa2cs and Bioinforma2cs Accelera2ng Scien2fic Discovery and Innova2on in Biomedical Research at NYULMC through Advanced Compu2ng Efstra'os Efstathiadis,
Chapter 3. Database Architectures and the Web Transparencies
Week 2: Chapter 3 Chapter 3 Database Architectures and the Web Transparencies Database Environment - Objec
Trinity Advanced Technology System Overview
Trinity Advanced Technology System Overview Manuel Vigil Trinity Project Director Douglas Doerfler Trinity Chief Architect 1 Outline ASC Compu/ng Strategy Project Drivers and Procurement Process Pla;orm
HPC and Big Data. EPCC The University of Edinburgh. Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk
HPC and Big Data EPCC The University of Edinburgh Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk EPCC Facilities Technology Transfer European Projects HPC Research Visitor Programmes Training
The Fusion of Supercomputing and Big Data. Peter Ungaro President & CEO
The Fusion of Supercomputing and Big Data Peter Ungaro President & CEO The Supercomputing Company Supercomputing Big Data Because some great things never change One other thing that hasn t changed. Cray
Mission Need Statement for the Next Generation High Performance Production Computing System Project (NERSC-8)
Mission Need Statement for the Next Generation High Performance Production Computing System Project () (Non-major acquisition project) Office of Advanced Scientific Computing Research Office of Science
Barry Bolding, Ph.D. VP, Cray Product Division
Barry Bolding, Ph.D. VP, Cray Product Division 1 Corporate Overview Trends in Supercomputing Types of Supercomputing and Cray s Approach The Cloud The Exascale Challenge Conclusion 2 Slide 3 Seymour Cray
Big Data Processing Experience in the ATLAS Experiment
Big Data Processing Experience in the ATLAS Experiment A. on behalf of the ATLAS Collabora5on Interna5onal Symposium on Grids and Clouds (ISGC) 2014 March 23-28, 2014 Academia Sinica, Taipei, Taiwan Introduction
2015-16 ITS Strategic Plan Enabling an Unbounded University
2015-16 ITS Strategic Plan Enabling an Unbounded University Update: July 31, 2015 IniAaAve: Agility Through Technology Vision Mission Enable Unbounded Learning Support student success through the innovaave
Cloud Compu?ng & Big Data in Higher Educa?on and Research: African Academic Experience
3 rd SG13 Regional Workshop for Africa on ITU- T Standardiza?on Challenges for Developing Countries Working for a Connected Africa (Livingstone, Zambia, 23-24 February 2015) Cloud Compu?ng & Big Data in
Kriterien für ein PetaFlop System
Kriterien für ein PetaFlop System Rainer Keller, HLRS :: :: :: Context: Organizational HLRS is one of the three national supercomputing centers in Germany. The national supercomputing centers are working
Solvers, Algorithms and Libraries (SAL)
Solvers, Algorithms and Libraries (SAL) D. Knoll (LANL), J. Wohlbier (LANL), M. Heroux (SNL), R. Hoekstra (SNL), S. Pautz (SNL), R. Hornung (LLNL), U. Yang (LLNL), P. Hovland (ANL), R. Mills (ORNL), S.
Jean-Pierre Panziera Teratec 2011
Technologies for the future HPC systems Jean-Pierre Panziera Teratec 2011 3 petaflop systems : TERA 100, CURIE & IFERC Tera100 Curie IFERC 1.25 PetaFlops 256 TB ory 30 PB disk storage 140 000+ Xeon cores
A R o a d t o y o u r C l o u d. Professional Service. C R M a n d C l o u d C o n s u l t i n g
RM-C A R o a d t o y o u r C l o u d Professional Service C R M a n d C l o u d C o n s u l t i n g CRM-C Highlights! A Unique Cloud CRM Consulting service firm! Specializing in cloud CRM and Office Collaboration
Predictive Community Computational Tools for Virtual Plasma Science Experiments
Predictive Community Computational Tools for Virtual Plasma Experiments J.-L. Vay, E. Esarey, A. Koniges Lawrence Berkeley National Laboratory J. Barnard, A. Friedman, D. Grote Lawrence Livermore National
Virident HGST Leading the Flash Pla6orm Transforma:on March 2014
Virident HGST Leading the Flash Pla6orm Transforma:on March 2014 www.virident.com Storage Technology Division Hard Drive Division Storage Technology Division www.virident.com ENTERPRISE 2014, Virident
Cray Gemini Interconnect. Technical University of Munich Parallel Programming Class of SS14 Denys Sobchyshak
Cray Gemini Interconnect Technical University of Munich Parallel Programming Class of SS14 Denys Sobchyshak Outline 1. Introduction 2. Overview 3. Architecture 4. Gemini Blocks 5. FMA & BTA 6. Fault tolerance
Scalus A)ribute Workshop. Paris, April 14th 15th
Scalus A)ribute Workshop Paris, April 14th 15th Content Mo=va=on, objec=ves, and constraints Scalus strategy Scenario and architectural views How the architecture works Mo=va=on for this MCITN Storage
Data Storage and Data Transfer. Dan Hitchcock Acting Associate Director, Advanced Scientific Computing Research
Data Storage and Data Transfer Dan Hitchcock Acting Associate Director, Advanced Scientific Computing Research Office of Science Organization 2 Data to Enable Scientific Discovery DOE SC Investments in
High Performance Compu2ng and High Performance Data: exploring the growing use of Supercomputers in Oil and Gas Explora2on & Produc2on
High Performance Compu2ng and High Performance Data: exploring the growing use of Supercomputers in Oil and Gas Explora2on & Produc2on Lesley Wyborn1, Ben Evans1, David Lescinsky2 and Clinton Foster2 16
Perspec'ves on SDN. Roadmap to SDN Workshop, LBL
Perspec'ves on SDN Roadmap to SDN Workshop, LBL Philip Papadopoulos San Diego Supercomputer Center California Ins8tute for Telecommunica8ons and Informa8on Technology University of California, San Diego
ARTIST Methodology and Tooling. Jesus Gorroñogoitia - Atos SOC Crete, 1 st July 2015
ARTIST Methodology and Tooling Jesus Gorroñogoitia - Atos SOC Crete, 1 st July 2015 Motivation: From SaaP to SaaS So#ware as a Product based Company So#ware as a Service based Company : Cloud Computing
New Jersey Big Data Alliance
Rutgers Discovery Informatics Institute (RDI 2 ) New Jersey s Center for Advanced Computation New Jersey Big Data Alliance Manish Parashar Director, Rutgers Discovery Informatics Institute (RDI 2 ) Professor,
PROJECT PORTFOLIO SUITE
ServiceNow So1ware Development manages Scrum or waterfall development efforts and defines the tasks required for developing and maintaining so[ware throughout the lifecycle, from incep4on to deployment.
Locales Domain Maps Layouts Distribu?ons Chapel Standard Layouts and Distribu?ons User defined Domain Maps. Chapel: Domain Maps
Locales Domain Maps Layouts Distribu?ons Chapel Standard Layouts and Distribu?ons User defined Domain Maps Chapel: Domain Maps 14 Domains are first class index sets Specify the size and shape of arrays
Science Gateways What are they and why are they having such a tremendous impact on science? Nancy Wilkins- Diehr wilkinsn@sdsc.edu
Science Gateways What are they and why are they having such a tremendous impact on science? Nancy Wilkins- Diehr wilkinsn@sdsc.edu What is a science gateway? science gateway /sī əәns gāt wā / n. 1. an
Return on Experience on Cloud Compu2ng Issues a stairway to clouds. Experts Workshop Nov. 21st, 2013
Return on Experience on Cloud Compu2ng Issues a stairway to clouds Experts Workshop Agenda InGeoCloudS SoCware Stack InGeoCloudS Elas2city and Scalability Elas2c File Server Elas2c Database Server Elas2c
EOFS Workshop Paris Sept, 2011. Lustre at exascale. Eric Barton. CTO Whamcloud, Inc. eeb@whamcloud.com. 2011 Whamcloud, Inc.
EOFS Workshop Paris Sept, 2011 Lustre at exascale Eric Barton CTO Whamcloud, Inc. eeb@whamcloud.com Agenda Forces at work in exascale I/O Technology drivers I/O requirements Software engineering issues
An Application-Aware Approach to Systems Support for Big Data
An Application-Aware Approach to Systems Support for Big Data Hong Jiang National Science Foundation & Department of Computer Science & Engineering University of Nebraska Lincoln ASAP 2013, June 5, 2013
Berkeley Institute for Data Science (BIDS)
Berkeley Institute for Data Science (BIDS) Saul Perlmutter & David Culler Astrophysics EECS University of California, Berkeley BEARS 2014 The Fourth Paradigm 1. Empirical + experimental 2. Theore5cal 3.
NA-ASC-128R-14-VOL.2-PN
L L N L - X X X X - X X X X X Advanced Simulation and Computing FY15 19 Program Notebook for Advanced Technology Development & Mitigation, Computational Systems and Software Environment and Facility Operations
Introduc)on & Mo)va)on
Introduc)on & Mo)va)on This document is a result of work by the perfsonar Project (hdp://www.perfsonar.net) and is licensed under CC BY- SA 4.0 (hdps://crea)vecommons.org/licenses/by- sa/4.0/). Event Presenter,
Mining the Cloud.! Building and supporting a Virtual Research Infrastructure!
! Mining the Cloud.! Building and supporting a Research Infrastructure! Gary Wroblewski Applica2on Coordinator/Mgr. Technical Services The Herbert H. and Grace A. Dow College of Health Professions Central
Cluster, Grid, Cloud Concepts
Cluster, Grid, Cloud Concepts Kalaiselvan.K Contents Section 1: Cluster Section 2: Grid Section 3: Cloud Cluster An Overview Need for a Cluster Cluster categorizations A computer cluster is a group of
Cray: Enabling Real-Time Discovery in Big Data
Cray: Enabling Real-Time Discovery in Big Data Discovery is the process of gaining valuable insights into the world around us by recognizing previously unknown relationships between occurrences, objects
Bulk Synchronous Programmers and Design
Linux Kernel Co-Scheduling For Bulk Synchronous Parallel Applications ROSS 2011 Tucson, AZ Terry Jones Oak Ridge National Laboratory 1 Managed by UT-Battelle Outline Motivation Approach & Research Design
BIG DATA AND INVESTIGATIVE ANALYTICS
The New Fron+er BIG DATA AND INVESTIGATIVE ANALYTICS A Publication of Infobright Table of Contents Introduc+on 3 Chapter 1: What Is Inves+ga+ve Analy+cs?. 4 Chapter 2: Top Five Requirements for Inves+ga+ve
Workflow Tools at NERSC. Debbie Bard djbard@lbl.gov NERSC Data and Analytics Services
Workflow Tools at NERSC Debbie Bard djbard@lbl.gov NERSC Data and Analytics Services NERSC User Meeting March 21st, 2016 What Does Workflow Software Do? Automate connec+on of applica+ons Chain together
www.xenon.com.au STORAGE HIGH SPEED INTERCONNECTS HIGH PERFORMANCE COMPUTING VISUALISATION GPU COMPUTING
www.xenon.com.au STORAGE HIGH SPEED INTERCONNECTS HIGH PERFORMANCE COMPUTING GPU COMPUTING VISUALISATION XENON Accelerating Exploration Mineral, oil and gas exploration is an expensive and challenging
Everything You Need to Know about Cloud BI. Freek Kamst
Everything You Need to Know about Cloud BI Freek Kamst Business Analy2cs Insight, Bussum June 10th, 2014 What s it all about? Has anything changed in the world of BI? Is Cloud Compu2ng a Hype or here to
Windows Server Performance Monitoring
Spot server problems before they are noticed The system s really slow today! How often have you heard that? Finding the solution isn t so easy. The obvious questions to ask are why is it running slowly
Outline. High Performance Computing (HPC) Big Data meets HPC. Case Studies: Some facts about Big Data Technologies HPC and Big Data converging
Outline High Performance Computing (HPC) Towards exascale computing: a brief history Challenges in the exascale era Big Data meets HPC Some facts about Big Data Technologies HPC and Big Data converging
Ellip%cal Mobile Solu%ons Micro- Modular Data Centers Are: The Next Generation Green Data Centers Solution
Ellip%cal Mobile Solu%ons Micro- Modular Data Centers Are: The Next Generation Green Data Centers Solution Copyright Ellip.cal Mobile Solu.ons 2009 www.ellip.calmobilesolu.ons.com 1 EMS CURRENT PRODUCTS
Better Transnational Access and Data Sharing to Solve Common Questions
Better Transnational Access and Data Sharing to Solve Common Questions Julia Lane American Ins0tutes for Research University of Strasbourg University of Melbourne Overview Common Ques0ons New kinds of
Modeling Big Data/HPC Storage Using Massively Parallel Simula:on
Modeling Big Data/HPC Storage Using Massively Parallel Simula:on Chris Carothers (CCNI) Misbah Mubarak (CS) Rensselaer Polytechnic Ins:tute chrisc@cs.rpi.edu Rob Ross Phil Carns MCS/ANL rross@mcs.anl.gov
How the ersa Problem became the ersa Solu3on. Why a network and network security is impera3ve for ersa s NeCTAR cloud. Paul Bartczak Infrastructure
How the ersa Problem became the ersa Solu3on. Why a network and network security is impera3ve for ersa s NeCTAR cloud. Paul Bartczak Infrastructure Manager About ersa eresearch SA is a collabora3ve joint
Data Intensive Science and Computing
DEFENSE LABORATORIES ACADEMIA TRANSFORMATIVE SCIENCE Efficient, effective and agile research system INDUSTRY Data Intensive Science and Computing Advanced Computing & Computational Sciences Division University
How To Make A Cloud Based Computer Power Available To A Computer (For Free)
Cloud Compu)ng Adam Belloum Ins)tute of Informa)cs University of Amsterdam a.s.z.belloum@uva.nl High Performance compu)ng Curriculum, Jan 2015 hgp://www.hpc.uva.nl/ UvA- SURFsara What is Cloud Compu)ng?
Denis Caromel, CEO Ac.veEon. Orchestrate and Accelerate Applica.ons. Open Source Cloud Solu.ons Hybrid Cloud: Private with Burst Capacity
Cloud computing et Virtualisation : applications au domaine de la Finance Denis Caromel, CEO Ac.veEon Orchestrate and Accelerate Applica.ons Open Source Cloud Solu.ons Hybrid Cloud: Private with Burst
DNS Big Data Analy@cs
Klik om de s+jl te bewerken Klik om de models+jlen te bewerken! Tweede niveau! Derde niveau! Vierde niveau DNS Big Data Analy@cs Vijfde niveau DNS- OARC Fall 2015 Workshop October 4th 2015 Maarten Wullink,
Dr. Raju Namburu Computational Sciences Campaign U.S. Army Research Laboratory. The Nation s Premier Laboratory for Land Forces UNCLASSIFIED
Dr. Raju Namburu Computational Sciences Campaign U.S. Army Research Laboratory 21 st Century Research Continuum Theory Theory embodied in computation Hypotheses tested through experiment SCIENTIFIC METHODS
Apache Hadoop: The Pla/orm for Big Data. Amr Awadallah CTO, Founder, Cloudera, Inc. aaa@cloudera.com, twicer: @awadallah
Apache Hadoop: The Pla/orm for Big Data Amr Awadallah CTO, Founder, Cloudera, Inc. aaa@cloudera.com, twicer: @awadallah 1 The Problems with Current Data Systems BI Reports + Interac7ve Apps RDBMS (aggregated
The Development of Cloud Interoperability
NSC- JST Workshop The Development of Cloud Interoperability Weicheng Huang Na7onal Center for High- performance Compu7ng Na7onal Applied Research Laboratories 1 Outline Where are we? Our experiences before
Cloudian The Storage Evolution to the Cloud.. Cloudian Inc. Pre Sales Engineering
Cloudian The Storage Evolution to the Cloud.. Cloudian Inc. Pre Sales Engineering Agenda Industry Trends Cloud Storage Evolu4on of Storage Architectures Storage Connec4vity redefined S3 Cloud Storage Use
Processing of Mix- Sensi0vity Video Surveillance Streams on Hybrid Clouds
Processing of Mix- Sensi0vity Video Surveillance Streams on Hybrid Clouds Chunwang Zhang, Ee- Chien Chang School of Compu2ng, Na2onal University of Singapore 28 th June, 2014 Outline 1. Mo0va0on 2. Hybrid
Next-Generation Networking for Science
Next-Generation Networking for Science ASCAC Presentation March 23, 2011 Program Managers Richard Carlson Thomas Ndousse Presentation