Modeling Big Data/HPC Storage Using Massively Parallel Simula:on

Size: px

Start display at page:

Download "Modeling Big Data/HPC Storage Using Massively Parallel Simula:on"

Jeremy Jacobs
7 years ago
Views:

1 Modeling Big Data/HPC Storage Using Massively Parallel Simula:on Chris Carothers (CCNI) Misbah Mubarak (CS) Rensselaer Polytechnic Ins:tute Rob Ross Phil Carns MCS/ANL Connec&ons of all the sub- networks in the world by Bill Cheswick, Lumeta Corp, 1998

2 Brief History of PDES and NM&S 1994: wireless PCS Network model. 32x32 square grid è 1024 LPs Mobile subscribers as events sent to grid LPs. No physical layer being modeled Performance: ~25K ev/sec on 8 DECsta:on worksta:ons 2003: Packet- level TCP over AT&T Backbone ~1 million packet- level TCP ﬂows over AT&T AS Leveraged Intel P4 HT, mul:- core processors Performance: 200K to 400K ev/sec on 4- way systems 2007: Slice- level BitTorrent Model over NBC Network Upto swarms of 256K Bi`orent clients with 2 to 16 seeders Consumed upwards of 48 GB Serial performance only: ~50K ev/sec 2007-Present saw a significant bump in PDES performance!

3 Blue Gene /P Layout In ~2009 ALCF/ANL Intrepid 163K cores/ 40 ~500 TFLOPS ~80TB RAM ~8 PB of disk over GPFS Custom OS kernel

4 NSF MRI Balanced CCNI Blue Gene/Q Phase 1: GF/wa` #1 on Green 500 list (architecture) 10PF and 20PF DOE systems Exec Model: 64K threads/16k cores 32 TB RAM 32 I/O nodes (4x over typical BG/Qs) RAM Storage Accelerator GB/sec GB each Disk storage TB disk Bandwidth: 5 to 24 GB/sec Viz systems CCNI: 16 servers w/ dual GPUs EMACS: display wall + servers WHAT CAN WE DO WITH THIS COMPUTE POWER FOR STORAGE M&S?

5 12.27 billion ev/sec for 10% remote on 65,536 cores!! ROSS is an op&mis&c Time Warp discrete- event simula&on engine designed for massively parallel systems 4 billion ev/sec for 100% remote on 65,536 cores!! Observed similar scaling on Blue Gene/ Q using 256K MPI ranks PHOLD benchmark on ROSS w/ 1 M 10 ev each!

6 CODES Project: CO- Design of Exascale Storage Top I/O job: Plasma Physics 67 TBs per job 10+ hrs to execute Over 2 hr idle period Overall, Bursty I/O! How to design an Exascale storage system? (e.g., Today that s ~1,000,000 3 TB hard disk system)

7 Modeling Every Level: e.g., File Open applica:on level request storage level Each box represents an event in the model! CIOD level file system level PVFS clients talk to PVFS servers itera:vely to find entry Randomly select a fileserver and create the object If we just model the file system complexity, what happens? 7

8 Model vs Data: Shared Unaligned Read Test Underscores the need for co- design with real experimental performance data!! Used POSIX interface Used 4MB (4 * 10^6) accesses for a total of 64MB per process Requests span mul:ple file stripe units, requiring that requests serviced by 2 storage node rather than one Simulated read performance curve is similar to simulated write performance curve because the striping algorithm is the same Max error is 30-40% Possible reasons: Lack of queuing at fileserver and Myricom network layer 8

9 The Dragonfly Network Topology A two level directly connected topology Uses high- radix routers Large number of ports per router Each port has moderate bandwidth p : Number of compute nodes connected to a router a : Number of routers in a group h : Number of global channels per router k=a + p + h 1 a=2p=2h (Recommended configura:on)

10 ROSS Dragonfly Performance Results on BG/P vs. BG/Q (for a 50 million node model) The event efficiency stays high on both BG/P and BG/Q as each MPI task has substan:al work load The computa:on performed at each MPI task dominates the number of rolled back events

11 Billion- Node Torus Network Model Using ROSS ROSS is a massively parallel discrete- event simulator Has scaled to 131,072 cores Yields very good strong scaling/extreme execu:on :me compression For accurate storage simula:ons, network is clearly important! So, can we model an exascale like network at the packet- level 32^6 (~1 billion) node Torus Topology consumes > 2 TB Small torus validated model against Blue Gene/L Torus network a b c d number of processors 4,096 8,192 16,384 32,768 65, ,072 efficiency 99.83% 99.90% 99.83% 99.55% 98.89% 97.51% event- rate (M/sec) 639 1,192 2,260 4,002 7,307 12,359 remote event percentage secondary rollback rate ( 10-5 ) 11.71% 12.39% 13.77% 16.53% 16.88% 17.22%

12 Summary & Forward Challenges 1. TAKE AWAY: Big Data/HPC storage systems can be effec:vely modeled using massively parallel simula:on tools and techniques ROSS is open source at available at our wiki site: odin.cs.rpi.edu 2. Need models and co- design around hybrid HPC/cloud storage systems. 3. Need power, failure and recovery models. 4. Need to exploit simula:on s out of band capabili:es. 5. Extend parallel simula:on engines for parallel I/O, data collec:on. 6. Massively parallel dynamic load balancing of models, especially under irregular big data/hpc workloads. 7. Make simula:on engines and models easier to use and configure. 8. Valida:on/Verifica:on techniques at scale when you don t have real experimental data.

Appro Supercomputer Solutions Best Practices Appro 2012 Deployment Successes. Anthony Kenisky, VP of North America Sales

Appro Supercomputer Solutions Best Practices Appro 2012 Deployment Successes Anthony Kenisky, VP of North America Sales About Appro Over 20 Years of Experience 1991 2000 OEM Server Manufacturer 2001-2007