Parallel I/O on Mira Venkat Vishwanath and Kevin Harms

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Parallel I/O on Mira Venkat Vishwanath and Kevin Harms"

Transcription

1 Parallel I/O on Mira Venkat Vishwanath and Kevin Harms Argonne Na*onal Laboratory

2 ALCF-2 I/O Infrastructure Mira BG/Q Compute Resource Tukey Analysis Cluster 48K Nodes 768K Cores 10 PFlops 768X 2 GB/s 1536 GB/s 384 I/O Nodes 1536 GB/s QDR IB Switch Complex 900+ ports 384 GB/s GB/s 6 TB Memory 96 Nodes 192 GPUs 128 DDN VM File Server 26 PB ~250GB/s Storage System Storage System consists of 16 DDN SFA12KE couplets with each couplet consisting of 560 x 3TB HDD, 32 x 200GB SSD, 8 Virtual Machines (VM) with two QDR IB ports per VM

3 I/O Subsystem of BG/Q- Simplified View 128 BG/Q Compute Nodes 2x2x4x4x2 2 GB/s 2 GB/s ION 4 GB/s For every 128 compute nodes, we have two nodes designated as Bridge nodes. Each Bridge node has a link (11 th ) to the I/O Node. On the ION, we have a QDR Infiniband connected on a PCI-e Gen2 Slot 3

4 Control and I/O Daemon (CIOD) I/O Forwarding mechanism for BG/P and BG/Q CIOD is the I/O forwarding infrastructure provided by IBM for the Blue Gene / P For each compute node, we have a dedicated I/O proxy process to handle the associated I/O operations On BG/Q, we have a thread based implementation instead of process-based

5 Intrepid vs Mira : I/O Comparison Intrepid BG/P Mira BG/Q Increase Flops PF 10 PF 20X Cores 160K 768K ~5X ION / CN ratio 1 / 64 1 / X IONs per Rack X Total Storage 6 PB 35 PB 6X I/O Throughput 62GB/s 240 GB/s Peak 4X User Quota? None Quotas enforced!! I/O per Flops (%) X - 5

6 I/O interfaces and Libraries on Mira Interfaces POSIX MPI- IO Higher- level Libraries HDF5 Netcdf4 PnetCDF I/O Performance on Mira Shared File Write ~70 GB/s Shared File Read ~180 GB/s 6

7 I/O Recommendations Single shared file vs file per process (MPI rank) Ideally, you should write a few large files Use MPI- IO Interfaces Use MPI Collec*ve I/O (eg. MPI_File_write_at_all()) when you are wri*ng small data sizes per rank qsub GPFS --env Block BGLOCKLESSMPIO_F_TYPE=0x size on Mira : 8 MB For non- overlapping writes pass the environment variable at job submission BGLOCKLESSMPIO_F_TYPE=0x or prefix file with bglockless:/ to disable locking in the ADIO layer Factor of two improvement in performance For POSIX, instead of lseek and write, use pwrite pwrite(int fd, const void *buf, size_t count, off_t offset); 7

8 More Tips and Tricks for I/O on BG/Q // Create and Preallocate the File On Master if ( 0 == m_rank) { retval = MPI_File_open(MPI_COMM_SELF, (char *)m_partfilename, Pre- create the files before wri*ng to them MPI_MODE_WRONLY MPI_MODE_CREATE, Increases shared file write performance by a factor MPI_INFO_NULL,&m_fileHandle); assert(retval == MPI_SUCCESS); Pre- allocate files Useful // Preallocate if you File know the exact size of data to be wrinen. MPI_File_set_size(m_fileHandle, m_partfilesize); Have a rank, say 0, create & allocate (MPI_File_set_size or MPI_File_close (&m_filehandle); pruncate) } and close. Next, have all ranks open this file for wri*ng. MPI_Barrier (MPI_COMM_WORLD); HDF5 Datatype // Open the File for Writing Instead retval = MPI_File_open(MPI_COMM_WORLD, of Na*ve, use Big Endian(eg. (char H5T_IEEE_F64BE) *)m_partfilename, MPI_MODE_WRONLY, MPI_INFO_NULL, &m_filehandle); Avoid Complex MPI Datatypes Keep assert(retval things == MPI_SUCCESS); simple in produc*on codes Data MPI_Barrier Compression (MPI_COMM_WORLD); 8

9 Tools for I/O Performance Profiling on BG/Q HPM Bob Walkup hnps:// guides/hpctw Darshan Phil Carns and Kevin Harms hnps:// guides/darshan TAU Sameer Shende hnps:// guides/tuning- and- analysis- u*li*es- tau opttrackio in TAU_OPTIONS 9

10 Data for MPI rank 0 of Profiling I/O using HPM Times and statistics from MPI_Init() to MPI_Finalize() MPI Routine #calls avg. bytes time(sec) Link applica*on with MPI_Comm_size MPI_Comm_rank /sop/perpools/hpctw/lib/libmpihpm_smp.a MPI_Isend MPI_Recv /bgsys/drivers/ppcfloor/bgpm/lib/libbgpm.a MPI_Probe MPI_Waitall MPI_Barrier MPI_Gather rw- r- - r- - MPI_Allgather 1 venkatv Performance May : mpi_profile rw- r- - r- - MPI_Allgatherv 1 venkatv Performance May : mpi_profile rw- r- - r- - MPI_Reduce 1 venkatv Performance May : mpi_profile MPI_Allreduce rw- r- - r- - 1 venkatv Performance 5505 May 20 09:29 mpi_profile Job produces the following output, among others MPI_File_close MPI_File_open MPI_File_read_at MPI_File_write_at_all total communication time = seconds. total elapsed time = seconds. total MPI-IO time = seconds. 10

11 Profiling I/O using HPM MPI_File_read_at #calls avg. bytes?me(sec) MPI_File_write_at_all Performance Issue: 15 Collective calls writing 0 bytes taking 10 secs. 11

12 Darshan An I/O Characterization Tool An open- source tool developed for sta*s*cal profiling of I/O Designed to be lightweight and low overhead Finite memory alloca*on for sta*s*cs (about 2MB) done during MPI_Init Overhead of 1-2% total to record I/O calls Varia*on of I/O is typically around 4-5% Darshan does not create detailed func*on call traces No source modifica*ons Uses PMPI interfaces to intercept MPI calls Use ld wrapping to intercept POSIX calls Can use dynamic linking with LD_PRELOAD instead Stores results in single compressed log file 12

13 fms c48l24 am3p5.x (3/27/2010) 1 of 2 Darshan on BG/Q uid: 6277 nprocs: 1944 runtime: 5382 seconds Percentage of run time Average I/O cost per process POSIX MPI-IO Read Write Metadata Other (including application compute) Ops (Total, All Processes) 4.5e+07 4e e+07 3e e+07 2e e+07 1e+07 5e+06 0 Read Write Open Stat Seek MmapFsync POSIX MPI-IO Indep. I/O Operation Counts MPI-IO Coll. logs: /projects/logs/darshan/vesta /projects/logs/darshan/mira bins: /soft/perftools/darshan/darshan/bin 3e+07 I/O Sizes 4e+07 I/O Pattern Count (Total, All Procs) 2.5e+07 2e e+07 1e+07 5e+06 Ops (Total, All Procs) 3.5e+07 3e e+07 2e e+07 1e+07 wrappers: /soft/perftools/darshan/darshan/ wrappers/<wrapper> 1K-10K 101-1K M-4M 100K-1M 10K-100K Read 1G+ 100M-1G 10M-100M 4M-10M Write Most Common Access Sizes access size count Read Total Sequential Write Consecutive export PATH=/soft/perftools/ darshan/darshan/wrappers/xl:$ {PATH} 0 5e+06./fms c48l24 am3p5.x <unknown args> 13

14 33 Early Experience Scaling I/O for HACC HACC on Mira: Early Science Test Run Hybrid/Hardware Accelerated First Large-Scale Computa*onal Simulation Cosmology on Mira code is lead by Salman 16 racks, 262,144 Habib cores at (one-third of Mira) Argonne 14 hours continuous, no checkpoint restarts Members include Katrin Simulation Parameters Heitmann, Hal 9.14 Finkel, Gpc simulation Adrian box Pope, Vitali Morozov 7 kpc force resolution 1.07 trillion particles Par*cle- in- Cell Science Motivations code Achieved ~14 Resolve PF on halos Sequoia that host and luminous red galaxies for 7 PF on Mira analysis of SDSS observations On Mira, HACC Cluster uses cosmology 1-10 Trillion par*cles Baryon acoustic oscillations in galaxy clustering Zoom-in visualization of the density field illustrating the global spatial dynamic range of the simulation -- approximately a million-to-one 14

15 Early Experience Scaling I/O for HACC Typical Checkpoint/Restart dataset is ~ TB Analysis output, wrinen more frequently, is ~1TB- 10TB Step 1: Default I/O mechanism using single shared file with MPI- IO Achieved ~15 GB/s Step 2: File per rank Too many files at 32K Nodes (262K files with 8RPN) Step 3: An I/O mechanism wri*ng 1 File per ION Topology- aware I/O to reduce network conten*on Achieves up to 160 GB/s (~10 X improvement) Wrinen and read ~10 PB of data on Mira (and coun*ng) 15

16 Questions 16

Running on Blue Gene/Q at Argonne Leadership Computing Facility (ALCF)

Running on Blue Gene/Q at Argonne Leadership Computing Facility (ALCF) Running on Blue Gene/Q at Argonne Leadership Computing Facility (ALCF) ALCF Resources: Machines & Storage Mira (Production) IBM Blue Gene/Q 49,152 nodes / 786,432 cores 768 TB of memory Peak flop rate:

More information

Parallel I/O on JUQUEEN

Parallel I/O on JUQUEEN Parallel I/O on JUQUEEN 3. February 2015 3rd JUQUEEN Porting and Tuning Workshop Sebastian Lührs, Kay Thust s.luehrs@fz-juelich.de, k.thust@fz-juelich.de Jülich Supercomputing Centre Overview Blue Gene/Q

More information

GPFS Storage Server. Concepts and Setup in Lemanicus BG/Q system" Christian Clémençon (EPFL-DIT)" " 4 April 2013"

GPFS Storage Server. Concepts and Setup in Lemanicus BG/Q system Christian Clémençon (EPFL-DIT)  4 April 2013 GPFS Storage Server Concepts and Setup in Lemanicus BG/Q system" Christian Clémençon (EPFL-DIT)" " Agenda" GPFS Overview" Classical versus GSS I/O Solution" GPFS Storage Server (GSS)" GPFS Native RAID

More information

Understanding and Improving Computational Science Storage Access through Continuous Characterization

Understanding and Improving Computational Science Storage Access through Continuous Characterization Understanding and Improving Computational Science Storage Access through Continuous Characterization Philip Carns, Kevin Harms, William Allcock, Charles Bacon, Samuel Lang, Robert Latham, and Robert Ross

More information

Portable, Scalable, and High-Performance I/O Forwarding on Massively Parallel Systems. Jason Cope copej@mcs.anl.gov

Portable, Scalable, and High-Performance I/O Forwarding on Massively Parallel Systems. Jason Cope copej@mcs.anl.gov Portable, Scalable, and High-Performance I/O Forwarding on Massively Parallel Systems Jason Cope copej@mcs.anl.gov Computation and I/O Performance Imbalance Leadership class computa:onal scale: >100,000

More information

Parallel file I/O bottlenecks and solutions

Parallel file I/O bottlenecks and solutions Mitglied der Helmholtz-Gemeinschaft Parallel file I/O bottlenecks and solutions Views to Parallel I/O: Hardware, Software, Application Challenges at Large Scale Introduction SIONlib Pitfalls, Darshan,

More information

LOAD BALANCING DISTRIBUTED OPERATING SYSTEMS, SCALABILITY, SS 2015. Hermann Härtig

LOAD BALANCING DISTRIBUTED OPERATING SYSTEMS, SCALABILITY, SS 2015. Hermann Härtig LOAD BALANCING DISTRIBUTED OPERATING SYSTEMS, SCALABILITY, SS 2015 Hermann Härtig ISSUES starting points independent Unix processes and block synchronous execution who does it load migration mechanism

More information

TECHNICAL GUIDELINES FOR APPLICANTS TO PRACE 7 th CALL (Tier-0)

TECHNICAL GUIDELINES FOR APPLICANTS TO PRACE 7 th CALL (Tier-0) TECHNICAL GUIDELINES FOR APPLICANTS TO PRACE 7 th CALL (Tier-0) Contributing sites and the corresponding computer systems for this call are: GCS@Jülich, Germany IBM Blue Gene/Q GENCI@CEA, France Bull Bullx

More information

Current Status of FEFS for the K computer

Current Status of FEFS for the K computer Current Status of FEFS for the K computer Shinji Sumimoto Fujitsu Limited Apr.24 2012 LUG2012@Austin Outline RIKEN and Fujitsu are jointly developing the K computer * Development continues with system

More information

Modeling Big Data/HPC Storage Using Massively Parallel Simula:on

Modeling Big Data/HPC Storage Using Massively Parallel Simula:on Modeling Big Data/HPC Storage Using Massively Parallel Simula:on Chris Carothers (CCNI) Misbah Mubarak (CS) Rensselaer Polytechnic Ins:tute chrisc@cs.rpi.edu Rob Ross Phil Carns MCS/ANL rross@mcs.anl.gov

More information

FLOW-3D Performance Benchmark and Profiling. September 2012

FLOW-3D Performance Benchmark and Profiling. September 2012 FLOW-3D Performance Benchmark and Profiling September 2012 Note The following research was performed under the HPC Advisory Council activities Participating vendors: FLOW-3D, Dell, Intel, Mellanox Compute

More information

Agenda. HPC Software Stack. HPC Post-Processing Visualization. Case Study National Scientific Center. European HPC Benchmark Center Montpellier PSSC

Agenda. HPC Software Stack. HPC Post-Processing Visualization. Case Study National Scientific Center. European HPC Benchmark Center Montpellier PSSC HPC Architecture End to End Alexandre Chauvin Agenda HPC Software Stack Visualization National Scientific Center 2 Agenda HPC Software Stack Alexandre Chauvin Typical HPC Software Stack Externes LAN Typical

More information

ANSYS FLUENT Performance Benchmark and Profiling. May 2009

ANSYS FLUENT Performance Benchmark and Profiling. May 2009 ANSYS FLUENT Performance Benchmark and Profiling May 29 Note The following research was performed under the HPC Advisory Council activities Participating vendors: AMD, ANSYS, Dell, Mellanox Compute resource

More information

NERSC File Systems and How to Use Them

NERSC File Systems and How to Use Them NERSC File Systems and How to Use Them David Turner! NERSC User Services Group! Joint Facilities User Forum on Data- Intensive Computing! June 18, 2014 The compute and storage systems 2014 Hopper: 1.3PF,

More information

High performance computing systems. Lab 1

High performance computing systems. Lab 1 High performance computing systems Lab 1 Dept. of Computer Architecture Faculty of ETI Gdansk University of Technology Paweł Czarnul For this exercise, study basic MPI functions such as: 1. for MPI management:

More information

Introduction History Design Blue Gene/Q Job Scheduler Filesystem Power usage Performance Summary Sequoia is a petascale Blue Gene/Q supercomputer Being constructed by IBM for the National Nuclear Security

More information

DIY Parallel Data Analysis

DIY Parallel Data Analysis I have had my results for a long time, but I do not yet know how I am to arrive at them. Carl Friedrich Gauss, 1777-1855 DIY Parallel Data Analysis APTESC Talk 8/6/13 Image courtesy pigtimes.com Tom Peterka

More information

Chao Chen 1 Michael Lang 2 Yong Chen 1. IEEE BigData, 2013. Department of Computer Science Texas Tech University

Chao Chen 1 Michael Lang 2 Yong Chen 1. IEEE BigData, 2013. Department of Computer Science Texas Tech University Chao Chen 1 Michael Lang 2 1 1 Data-Intensive Scalable Laboratory Department of Computer Science Texas Tech University 2 Los Alamos National Laboratory IEEE BigData, 2013 Outline 1 2 3 4 Outline 1 2 3

More information

LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance

LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance 11 th International LS-DYNA Users Conference Session # LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance Gilad Shainer 1, Tong Liu 2, Jeff Layton 3, Onur Celebioglu

More information

Session 2: MUST. Correctness Checking

Session 2: MUST. Correctness Checking Center for Information Services and High Performance Computing (ZIH) Session 2: MUST Correctness Checking Dr. Matthias S. Müller (RWTH Aachen University) Tobias Hilbrich (Technische Universität Dresden)

More information

Present and Future Computing Requirements for Computational Cosmology

Present and Future Computing Requirements for Computational Cosmology Present and Future Computing Requirements for Computational Cosmology 1 Presenter: Salman Habib Argonne National Laboratory Cosmic Frontier Computing Collaboration Computational Cosmology SciDAC-3 Project

More information

LS-DYNA Performance Benchmark and Profiling. February 2014

LS-DYNA Performance Benchmark and Profiling. February 2014 LS-DYNA Performance Benchmark and Profiling February 2014 Note The following research was performed under the HPC Advisory Council activities Participating vendors: Intel, Dell, Mellanox, LSTC Compute

More information

ECLIPSE Performance Benchmarks and Profiling. January 2009

ECLIPSE Performance Benchmarks and Profiling. January 2009 ECLIPSE Performance Benchmarks and Profiling January 2009 Note The following research was performed under the HPC Advisory Council activities AMD, Dell, Mellanox, Schlumberger HPC Advisory Council Cluster

More information

Lecture 31: Introduction to Parallel I/O. William Gropp

Lecture 31: Introduction to Parallel I/O. William Gropp Lecture 31: Introduction to Parallel I/O William Gropp www.cs.illinois.edu/~wgropp I/O and File Systems Most applications need persistent storage Typical to store persistent data in files, accessed through

More information

What is High Performance Computing?

What is High Performance Computing? What is High Performance Computing? V. Sundararajan Scientific and Engineering Computing Group Centre for Development of Advanced Computing Pune 411 007 vsundar@cdac.in Plan Why HPC? Parallel Architectures

More information

Lightning Introduction to MPI Programming

Lightning Introduction to MPI Programming Lightning Introduction to MPI Programming May, 2015 What is MPI? Message Passing Interface A standard, not a product First published 1994, MPI-2 published 1997 De facto standard for distributed-memory

More information

MPI Application Development Using the Analysis Tool MARMOT

MPI Application Development Using the Analysis Tool MARMOT MPI Application Development Using the Analysis Tool MARMOT HLRS High Performance Computing Center Stuttgart Allmandring 30 D-70550 Stuttgart http://www.hlrs.de 24.02.2005 1 Höchstleistungsrechenzentrum

More information

System Software for High Performance Computing. Joe Izraelevitz

System Software for High Performance Computing. Joe Izraelevitz System Software for High Performance Computing Joe Izraelevitz Agenda Overview of Supercomputers Blue Gene/Q System LoadLeveler Job Scheduler General Parallel File System HPC at UR What is a Supercomputer?

More information

OpenFOAM Performance Benchmark and Profiling. Jan 2010

OpenFOAM Performance Benchmark and Profiling. Jan 2010 OpenFOAM Performance Benchmark and Profiling Jan 2010 Note The following research was performed under the HPC Advisory Council activities Participating vendors: AMD, Dell, Mellanox Compute resource - HPC

More information

RA MPI Compilers Debuggers Profiling. March 25, 2009

RA MPI Compilers Debuggers Profiling. March 25, 2009 RA MPI Compilers Debuggers Profiling March 25, 2009 Examples and Slides To download examples on RA 1. mkdir class 2. cd class 3. wget http://geco.mines.edu/workshop/class2/examples/examples.tgz 4. tar

More information

Big Data in HPC Applications and Programming Abstractions. Saba Sehrish Oct 3, 2012

Big Data in HPC Applications and Programming Abstractions. Saba Sehrish Oct 3, 2012 Big Data in HPC Applications and Programming Abstractions Saba Sehrish Oct 3, 2012 Big Data in Computational Science - Size Data requirements for select 2012 INCITE applications at ALCF (BG/P) On-line

More information

Performance Monitoring of Parallel Scientific Applications

Performance Monitoring of Parallel Scientific Applications Performance Monitoring of Parallel Scientific Applications Abstract. David Skinner National Energy Research Scientific Computing Center Lawrence Berkeley National Laboratory This paper introduces an infrastructure

More information

Can High-Performance Interconnects Benefit Memcached and Hadoop?

Can High-Performance Interconnects Benefit Memcached and Hadoop? Can High-Performance Interconnects Benefit Memcached and Hadoop? D. K. Panda and Sayantan Sur Network-Based Computing Laboratory Department of Computer Science and Engineering The Ohio State University,

More information

Linux and High-Performance Computing

Linux and High-Performance Computing Linux and High-Performance Computing Outline Architectures & Performance Measurement Linux on High-Performance Computers Beowulf Clusters, ROCKS Kitten: A Linux-derived LWK Linux on I/O and Service Nodes

More information

OpenMP Programming on ScaleMP

OpenMP Programming on ScaleMP OpenMP Programming on ScaleMP Dirk Schmidl schmidl@rz.rwth-aachen.de Rechen- und Kommunikationszentrum (RZ) MPI vs. OpenMP MPI distributed address space explicit message passing typically code redesign

More information

Introduction to grid technologies, parallel and cloud computing. Alaa Osama Allam Saida Saad Mohamed Mohamed Ibrahim Gaber

Introduction to grid technologies, parallel and cloud computing. Alaa Osama Allam Saida Saad Mohamed Mohamed Ibrahim Gaber Introduction to grid technologies, parallel and cloud computing Alaa Osama Allam Saida Saad Mohamed Mohamed Ibrahim Gaber OUTLINES Grid Computing Parallel programming technologies (MPI- Open MP-Cuda )

More information

Mellanox Cloud and Database Acceleration Solution over Windows Server 2012 SMB Direct

Mellanox Cloud and Database Acceleration Solution over Windows Server 2012 SMB Direct Mellanox Cloud and Database Acceleration Solution over Windows Server 2012 Direct Increased Performance, Scaling and Resiliency July 2012 Motti Beck, Director, Enterprise Market Development Motti@mellanox.com

More information

Appro Supercomputer Solutions Best Practices Appro 2012 Deployment Successes. Anthony Kenisky, VP of North America Sales

Appro Supercomputer Solutions Best Practices Appro 2012 Deployment Successes. Anthony Kenisky, VP of North America Sales Appro Supercomputer Solutions Best Practices Appro 2012 Deployment Successes Anthony Kenisky, VP of North America Sales About Appro Over 20 Years of Experience 1991 2000 OEM Server Manufacturer 2001-2007

More information

Elements of Scalable Data Analysis and Visualization

Elements of Scalable Data Analysis and Visualization Elements of Scalable Data Analysis and Visualization www.ultravis.org DOE CGF Petascale Computing Session Tom Peterka tpeterka@mcs.anl.gov Mathematics and Computer Science Division 0. Preface - Science

More information

Reliability of NVM devices for I/O Acceleration on Supercomputing Systems

Reliability of NVM devices for I/O Acceleration on Supercomputing Systems Reliability of NVM devices for I/O Acceleration on Supercomputing Systems Hitoshi Sato* 1, Shuichi Ihara 2), Satoshi Matsuoka 1) *1 Tokyo Institute of Technology * 2 Data Direct Networks Japan Requirements

More information

Lecture 6: Introduction to MPI programming. Lecture 6: Introduction to MPI programming p. 1

Lecture 6: Introduction to MPI programming. Lecture 6: Introduction to MPI programming p. 1 Lecture 6: Introduction to MPI programming Lecture 6: Introduction to MPI programming p. 1 MPI (message passing interface) MPI is a library standard for programming distributed memory MPI implementation(s)

More information

Abaqus Performance Benchmark and Profiling. March 2015

Abaqus Performance Benchmark and Profiling. March 2015 Abaqus 6.14-2 Performance Benchmark and Profiling March 2015 2 Note The following research was performed under the HPC Advisory Council activities Special thanks for: HP, Mellanox For more information

More information

Petascale Software Challenges. Piyush Chaudhary piyushc@us.ibm.com High Performance Computing

Petascale Software Challenges. Piyush Chaudhary piyushc@us.ibm.com High Performance Computing Petascale Software Challenges Piyush Chaudhary piyushc@us.ibm.com High Performance Computing Fundamental Observations Applications are struggling to realize growth in sustained performance at scale Reasons

More information

PCI Express Impact on Storage Architectures and Future Data Centers. Ron Emerick, Oracle Corporation

PCI Express Impact on Storage Architectures and Future Data Centers. Ron Emerick, Oracle Corporation PCI Express Impact on Storage Architectures and Future Data Centers Ron Emerick, Oracle Corporation SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies

More information

PERFORMANCE OF PARALLEL IO ON LUSTRE AND GPFS

PERFORMANCE OF PARALLEL IO ON LUSTRE AND GPFS PERFORMANCE OF PARALLEL IO ON LUSTRE AND GPFS David Henty and Adrian Jackson (EPCC, The University of Edinburgh) Charles Moulinec and Vendel Szeremi (STFC, Daresbury Laboratory Outline Parallel IO problem

More information

ECLIPSE Best Practices Performance, Productivity, Efficiency. March 2009

ECLIPSE Best Practices Performance, Productivity, Efficiency. March 2009 ECLIPSE Best Practices Performance, Productivity, Efficiency March 29 ECLIPSE Performance, Productivity, Efficiency The following research was performed under the HPC Advisory Council activities HPC Advisory

More information

Advanced Techniques with Newton. Gerald Ragghianti Advanced Newton workshop Sept. 22, 2011

Advanced Techniques with Newton. Gerald Ragghianti Advanced Newton workshop Sept. 22, 2011 Advanced Techniques with Newton Gerald Ragghianti Advanced Newton workshop Sept. 22, 2011 Workshop Goals Gain independence Executing your work Finding Information Fixing Problems Optimizing Effectiveness

More information

JUROPA Linux Cluster An Overview. 19 May 2014 Ulrich Detert

JUROPA Linux Cluster An Overview. 19 May 2014 Ulrich Detert Mitglied der Helmholtz-Gemeinschaft JUROPA Linux Cluster An Overview 19 May 2014 Ulrich Detert JuRoPA JuRoPA Jülich Research on Petaflop Architectures Bull, Sun, ParTec, Intel, Mellanox, Novell, FZJ JUROPA

More information

EXPLOITING SHARED MEMORY TO IMPROVE PARALLEL I/O PERFORMANCE

EXPLOITING SHARED MEMORY TO IMPROVE PARALLEL I/O PERFORMANCE EXPLOITING SHARED MEMORY TO IMPROVE PARALLEL I/O PERFORMANCE Andrew B. Hastings Sun Microsystems, Inc. Alok Choudhary Northwestern University September 19, 2006 This material is based on work supported

More information

Storage Systems: 2014 and beyond. Jason Hick! Storage Systems Group!! NERSC User Group Meeting! February 6, 2014

Storage Systems: 2014 and beyond. Jason Hick! Storage Systems Group!! NERSC User Group Meeting! February 6, 2014 Storage Systems: 2014 and beyond Jason Hick! Storage Systems Group!! NERSC User Group Meeting! February 6, 2014 The compute and storage systems 2013 Hopper: 1.3PF, 212 TB RAM 2.2 PB Local Scratch 70 GB/s

More information

Massively Parallel Loading

Massively Parallel Loading Mitglied der Helmholtz-Gemeinschaft Massively Parallel Loading June 14 th 2013 27th International Conference on Supercomputing, Eugene, OR * ** Jülich Supercomputing Centre *** German Research School for

More information

Cluster Implementation and Management; Scheduling

Cluster Implementation and Management; Scheduling Cluster Implementation and Management; Scheduling CPS343 Parallel and High Performance Computing Spring 2013 CPS343 (Parallel and HPC) Cluster Implementation and Management; Scheduling Spring 2013 1 /

More information

Overlapping Data Transfer With Application Execution on Clusters

Overlapping Data Transfer With Application Execution on Clusters Overlapping Data Transfer With Application Execution on Clusters Karen L. Reid and Michael Stumm reid@cs.toronto.edu stumm@eecg.toronto.edu Department of Computer Science Department of Electrical and Computer

More information

Correlating Multiple TB of Performance Data to User Jobs

Correlating Multiple TB of Performance Data to User Jobs Michael Kluge, ZIH Correlating Multiple TB of Performance Data to User Jobs Lustre User Group 2015, Denver, Colorado Zellescher Weg 12 Willers-Bau A 208 Tel. +49 351-463 34217 Michael Kluge (michael.kluge@tu-dresden.de)

More information

PCIe Over Cable Provides Greater Performance for Less Cost for High Performance Computing (HPC) Clusters. from One Stop Systems (OSS)

PCIe Over Cable Provides Greater Performance for Less Cost for High Performance Computing (HPC) Clusters. from One Stop Systems (OSS) PCIe Over Cable Provides Greater Performance for Less Cost for High Performance Computing (HPC) Clusters from One Stop Systems (OSS) PCIe Over Cable PCIe provides greater performance 8 7 6 5 GBytes/s 4

More information

Data Deduplication in a Hybrid Architecture for Improving Write Performance

Data Deduplication in a Hybrid Architecture for Improving Write Performance Data Deduplication in a Hybrid Architecture for Improving Write Performance Data-intensive Salable Computing Laboratory Department of Computer Science Texas Tech University Lubbock, Texas June 10th, 2013

More information

Cosmological Simulations on Large, Heterogeneous Supercomputers

Cosmological Simulations on Large, Heterogeneous Supercomputers Cosmological Simulations on Large, Heterogeneous Supercomputers Adrian Pope (LANL) Cosmology on the Beach January 10, 2011 Slide 1 People LANL: Jim Ahrens, Lee Ankeny, Suman Bhattacharya, David Daniel,

More information

In situ data analysis and I/O acceleration of FLASH astrophysics simulation on leadership-class system using GLEAN

In situ data analysis and I/O acceleration of FLASH astrophysics simulation on leadership-class system using GLEAN In situ data analysis and I/O acceleration of FLASH astrophysics simulation on leadership-class system using GLEAN Venkatram Vishwanath 1, Mark Hereld 1, Michael E. Papka 1, Randy Hudson 2, G. Cal Jordan

More information

THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING. José Daniel García Sánchez ARCOS Group University Carlos III of Madrid

THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING. José Daniel García Sánchez ARCOS Group University Carlos III of Madrid THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING José Daniel García Sánchez ARCOS Group University Carlos III of Madrid Contents 2 The ARCOS Group. Expand motivation. Expand

More information

PADS GPFS Filesystem: Crash Root Cause Analysis. Computation Institute

PADS GPFS Filesystem: Crash Root Cause Analysis. Computation Institute PADS GPFS Filesystem: Crash Root Cause Analysis Computation Institute Argonne National Laboratory Table of Contents Purpose 1 Terminology 2 Infrastructure 4 Timeline of Events 5 Background 5 Corruption

More information

Hadoop on the Gordon Data Intensive Cluster

Hadoop on the Gordon Data Intensive Cluster Hadoop on the Gordon Data Intensive Cluster Amit Majumdar, Scientific Computing Applications Mahidhar Tatineni, HPC User Services San Diego Supercomputer Center University of California San Diego Dec 18,

More information

Accelerating Real Time Big Data Applications. PRESENTATION TITLE GOES HERE Bob Hansen

Accelerating Real Time Big Data Applications. PRESENTATION TITLE GOES HERE Bob Hansen Accelerating Real Time Big Data Applications PRESENTATION TITLE GOES HERE Bob Hansen Apeiron Data Systems Apeiron is developing a VERY high performance Flash storage system that alters the economics of

More information

Performance Characteristics of Large SMP Machines

Performance Characteristics of Large SMP Machines Performance Characteristics of Large SMP Machines Dirk Schmidl, Dieter an Mey, Matthias S. Müller schmidl@rz.rwth-aachen.de Rechen- und Kommunikationszentrum (RZ) Agenda Investigated Hardware Kernel Benchmark

More information

Object storage in Cloud Computing and Embedded Processing

Object storage in Cloud Computing and Embedded Processing Object storage in Cloud Computing and Embedded Processing Jan Jitze Krol Systems Engineer DDN We Accelerate Information Insight DDN is a Leader in Massively Scalable Platforms and Solutions for Big Data

More information

HPCC - Hrothgar Getting Started User Guide MPI Programming

HPCC - Hrothgar Getting Started User Guide MPI Programming HPCC - Hrothgar Getting Started User Guide MPI Programming High Performance Computing Center Texas Tech University HPCC - Hrothgar 2 Table of Contents 1. Introduction... 3 2. Setting up the environment...

More information

Weather Research and Forecasting (WRF) Performance Benchmark and Profiling. June 2015

Weather Research and Forecasting (WRF) Performance Benchmark and Profiling. June 2015 Weather Research and Forecasting (WRF) Performance Benchmark and Profiling June 2015 2 Note The following research was performed under the HPC Advisory Council activities Participating vendors: Intel,

More information

Performance Analysis of Mixed Distributed Filesystem Workloads

Performance Analysis of Mixed Distributed Filesystem Workloads Performance Analysis of Mixed Distributed Filesystem Workloads Esteban Molina-Estolano, Maya Gokhale, Carlos Maltzahn, John May, John Bent, Scott Brandt Motivation Hadoop-tailored filesystems (e.g. CloudStore)

More information

HETEROGENEOUS HPC, ARCHITECTURE OPTIMIZATION, AND NVLINK

HETEROGENEOUS HPC, ARCHITECTURE OPTIMIZATION, AND NVLINK HETEROGENEOUS HPC, ARCHITECTURE OPTIMIZATION, AND NVLINK Steve Oberlin CTO, Accelerated Computing US to Build Two Flagship Supercomputers SUMMIT SIERRA Partnership for Science 100-300 PFLOPS Peak Performance

More information

LS DYNA Performance Benchmarks and Profiling. January 2009

LS DYNA Performance Benchmarks and Profiling. January 2009 LS DYNA Performance Benchmarks and Profiling January 2009 Note The following research was performed under the HPC Advisory Council activities AMD, Dell, Mellanox HPC Advisory Council Cluster Center The

More information

PETASCALE DATA STORAGE INSTITUTE. SciDAC @ Petascale storage issues. 3 universities, 5 labs, G. Gibson, CMU, PI

PETASCALE DATA STORAGE INSTITUTE. SciDAC @ Petascale storage issues. 3 universities, 5 labs, G. Gibson, CMU, PI PETASCALE DATA STORAGE INSTITUTE 3 universities, 5 labs, G. Gibson, CMU, PI SciDAC @ Petascale storage issues www.pdsi-scidac.org Community building: ie. PDSW-SC07 (Sun 11th) APIs & standards: ie., Parallel

More information

Scalable Cloud Computing Solutions for Next Generation Sequencing Data

Scalable Cloud Computing Solutions for Next Generation Sequencing Data Scalable Cloud Computing Solutions for Next Generation Sequencing Data Matti Niemenmaa 1, Aleksi Kallio 2, André Schumacher 1, Petri Klemelä 2, Eija Korpelainen 2, and Keijo Heljanko 1 1 Department of

More information

Application Performance Characterization and Analysis on Blue Gene/Q

Application Performance Characterization and Analysis on Blue Gene/Q Application Performance Characterization and Analysis on Blue Gene/Q Bob Walkup (walkup@us.ibm.com) Click to add text 2008 Blue Gene/Q : Power-Efficient Computing System date GHz cores/rack largest-system

More information

VMware Virtual SAN Hardware Guidance. TECHNICAL MARKETING DOCUMENTATION v 1.0

VMware Virtual SAN Hardware Guidance. TECHNICAL MARKETING DOCUMENTATION v 1.0 VMware Virtual SAN Hardware Guidance TECHNICAL MARKETING DOCUMENTATION v 1.0 Table of Contents Introduction.... 3 Server Form Factors... 3 Rackmount.... 3 Blade.........................................................................3

More information

Kommunikation in HPC-Clustern

Kommunikation in HPC-Clustern Kommunikation in HPC-Clustern Communication/Computation Overlap in MPI W. Rehm and T. Höfler Department of Computer Science TU Chemnitz http://www.tu-chemnitz.de/informatik/ra 11.11.2005 Outline 1 2 Optimize

More information

BG/Q Performance Tools. Sco$ Parker BG/Q Early Science Workshop: March 19-21, 2012 Argonne Leadership CompuGng Facility

BG/Q Performance Tools. Sco$ Parker BG/Q Early Science Workshop: March 19-21, 2012 Argonne Leadership CompuGng Facility BG/Q Performance Tools Sco$ Parker BG/Q Early Science Workshop: March 19-21, 2012 BG/Q Performance Tool Development In conjuncgon with the Early Science program an Early SoMware efforts was inigated to

More information

Storage Architectures. Ron Emerick, Oracle Corporation

Storage Architectures. Ron Emerick, Oracle Corporation PCI Express PRESENTATION and Its TITLE Interfaces GOES HERE to Flash Storage Architectures Ron Emerick, Oracle Corporation SNIA Legal Notice The material contained in this tutorial is copyrighted by the

More information

BG/Q Performance Tools. Sco$ Parker Leap to Petascale Workshop: May 22-25, 2012 Argonne Leadership CompuCng Facility

BG/Q Performance Tools. Sco$ Parker Leap to Petascale Workshop: May 22-25, 2012 Argonne Leadership CompuCng Facility BG/Q Performance Tools Sco$ Parker Leap to Petascale Workshop: May 22-25, 2012 BG/Q Performance Tool Development In conjunccon with the Early Science program an Early SoIware efforts was inicated to bring

More information

Software challenges in the implementation of large surveys: the case of J-PAS

Software challenges in the implementation of large surveys: the case of J-PAS Software challenges in the implementation of large surveys: the case of J-PAS 1/21 Paulo Penteado - IAG/USP pp.penteado@gmail.com http://www.ppenteado.net/ast/pp_lsst_201204.pdf (K. Taylor) (A. Fernández-Soto)

More information

Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components

Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components of Hadoop. We will see what types of nodes can exist in a Hadoop

More information

Beyond Embarrassingly Parallel Big Data. William Gropp www.cs.illinois.edu/~wgropp

Beyond Embarrassingly Parallel Big Data. William Gropp www.cs.illinois.edu/~wgropp Beyond Embarrassingly Parallel Big Data William Gropp www.cs.illinois.edu/~wgropp Messages Big is big Data driven is an important area, but not all data driven problems are big data (despite current hype).

More information

A Survey of Shared File Systems

A Survey of Shared File Systems Technical Paper A Survey of Shared File Systems Determining the Best Choice for your Distributed Applications A Survey of Shared File Systems A Survey of Shared File Systems Table of Contents Introduction...

More information

HP ProLiant DL580 Gen8 and HP LE PCIe Workload WHITE PAPER Accelerator 90TB Microsoft SQL Server Data Warehouse Fast Track Reference Architecture

HP ProLiant DL580 Gen8 and HP LE PCIe Workload WHITE PAPER Accelerator 90TB Microsoft SQL Server Data Warehouse Fast Track Reference Architecture WHITE PAPER HP ProLiant DL580 Gen8 and HP LE PCIe Workload WHITE PAPER Accelerator 90TB Microsoft SQL Server Data Warehouse Fast Track Reference Architecture Based on Microsoft SQL Server 2014 Data Warehouse

More information

Performance Analysis of a Cardiac Simulation Code Using IPM

Performance Analysis of a Cardiac Simulation Code Using IPM Performance Analysis of a Cardiac Simulation Code Using IPM Peter Strazdins* Computer Systems Group, Research School of Computer Science, Markus Hegland, Mathematical Sciences Institute, The Australian

More information

SR-IOV In High Performance Computing

SR-IOV In High Performance Computing SR-IOV In High Performance Computing Hoot Thompson & Dan Duffy NASA Goddard Space Flight Center Greenbelt, MD 20771 hoot@ptpnow.com daniel.q.duffy@nasa.gov www.nccs.nasa.gov Focus on the research side

More information

A Close Look at PCI Express SSDs. Shirish Jamthe Director of System Engineering Virident Systems, Inc. August 2011

A Close Look at PCI Express SSDs. Shirish Jamthe Director of System Engineering Virident Systems, Inc. August 2011 A Close Look at PCI Express SSDs Shirish Jamthe Director of System Engineering Virident Systems, Inc. August 2011 Macro Datacenter Trends Key driver: Information Processing Data Footprint (PB) CAGR: 100%

More information

PCI Express IO Virtualization Overview

PCI Express IO Virtualization Overview Ron Emerick, Oracle Corporation Author: Ron Emerick, Oracle Corporation SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and

More information

Cosmological Simulations on Large, Accelerated Supercomputers

Cosmological Simulations on Large, Accelerated Supercomputers Cosmological Simulations on Large, Accelerated Supercomputers Adrian Pope (LANL) ICCS Workshop 27 January 2011 Slide 1 People LANL: Jim Ahrens, Lee Ankeny, Suman Bhattacharya, David Daniel, Pat Fasel,

More information

Mixing Hadoop and HPC Workloads on Parallel Filesystems

Mixing Hadoop and HPC Workloads on Parallel Filesystems Mixing Hadoop and HPC Workloads on Parallel Filesystems Esteban Molina-Estolano *, Maya Gokhale, Carlos Maltzahn *, John May, John Bent, Scott Brandt * * UC Santa Cruz, ISSDM, PDSI Lawrence Livermore National

More information

Use of Hadoop File System for Nuclear Physics Analyses in STAR

Use of Hadoop File System for Nuclear Physics Analyses in STAR 1 Use of Hadoop File System for Nuclear Physics Analyses in STAR EVAN SANGALINE UC DAVIS Motivations 2 Data storage a key component of analysis requirements Transmission and storage across diverse resources

More information

Unstructured Data Accelerator (UDA) Author: Motti Beck, Mellanox Technologies Date: March 27, 2012

Unstructured Data Accelerator (UDA) Author: Motti Beck, Mellanox Technologies Date: March 27, 2012 Unstructured Data Accelerator (UDA) Author: Motti Beck, Mellanox Technologies Date: March 27, 2012 1 Market Trends Big Data Growing technology deployments are creating an exponential increase in the volume

More information

FUJITSU x86 HPC Cluster

FUJITSU x86 HPC Cluster Your Gateway to HPC simplicity FUJITSU x86 HPC Cluster 0 FUJITSU : PRIMERGY and CELSIUS Intermediate Cover Subtitle 1 Fujitsu x86 Server Scale Up / SMP Computing Exhibit in the booth PRIMERGY CX400 S1

More information

Libmonitor: A Tool for First-Party Monitoring

Libmonitor: A Tool for First-Party Monitoring Libmonitor: A Tool for First-Party Monitoring Mark W. Krentel Dept. of Computer Science Rice University 6100 Main St., Houston, TX 77005 krentel@rice.edu ABSTRACT Libmonitor is a library that provides

More information

Architecting a High Performance Storage System

Architecting a High Performance Storage System WHITE PAPER Intel Enterprise Edition for Lustre* Software High Performance Data Division Architecting a High Performance Storage System January 2014 Contents Introduction... 1 A Systematic Approach to

More information

HITACHI VIRTUAL STORAGE PLATFORM FAMILY MATRIX

HITACHI VIRTUAL STORAGE PLATFORM FAMILY MATRIX HITACHI VIRTUAL STORAGE PLATFORM FAMILY MATRIX G1000 Capacity Specifications Maximum (Max.) Number of Hard Drives, Including Spares 264 SFF 252 LFF 480 SFF 480 LFF 720 SFF 720 LFF 1,440 SFF 1,440 LFF 2,304

More information

Performance Characteristics of VMFS and RDM VMware ESX Server 3.0.1

Performance Characteristics of VMFS and RDM VMware ESX Server 3.0.1 Performance Study Performance Characteristics of and RDM VMware ESX Server 3.0.1 VMware ESX Server offers three choices for managing disk access in a virtual machine VMware Virtual Machine File System

More information

www.thinkparq.com www.beegfs.com

www.thinkparq.com www.beegfs.com www.thinkparq.com www.beegfs.com KEY ASPECTS Maximum Flexibility Maximum Scalability BeeGFS supports a wide range of Linux distributions such as RHEL/Fedora, SLES/OpenSuse or Debian/Ubuntu as well as a

More information

HPC Update: Engagement Model

HPC Update: Engagement Model HPC Update: Engagement Model MIKE VILDIBILL Director, Strategic Engagements Sun Microsystems mikev@sun.com Our Strategy Building a Comprehensive HPC Portfolio that Delivers Differentiated Customer Value

More information

Enabling a Strict Consistency Semantics Model in Parallel HDF5

Enabling a Strict Consistency Semantics Model in Parallel HDF5 Enabling a Strict Consistency Semantics Model in Parallel HDF5 Introduced with HDF5 Release 1.8.9 in May 2012 http://www.hdfgroup.org Copyright Notice and License Terms for HDF5 (Hierarchical Data Format

More information

Amadeus SAS Specialists Prove Fusion iomemory a Superior Analysis Accelerator

Amadeus SAS Specialists Prove Fusion iomemory a Superior Analysis Accelerator WHITE PAPER Amadeus SAS Specialists Prove Fusion iomemory a Superior Analysis Accelerator 951 SanDisk Drive, Milpitas, CA 95035 www.sandisk.com SAS 9 Preferred Implementation Partner tests a single Fusion

More information

Data Management in the Cloud: Limitations and Opportunities. Annies Ductan

Data Management in the Cloud: Limitations and Opportunities. Annies Ductan Data Management in the Cloud: Limitations and Opportunities Annies Ductan Discussion Outline: Introduc)on Overview Vision of Cloud Compu8ng Managing Data in The Cloud Cloud Characteris8cs Data Management

More information