HPC Growing Pains. Lessons learned from building a Top500 supercomputer

Transcription

1 HPC Growing Pains Lessons learned from building a Top500 supercomputer John L. Wofford Center for Computational Biology & Bioinformatics Columbia University

2 I. What is C2B2? Outline Lessons learned from building a Top500 supercomputer: II. III. Before & After: Introducing Titan, our 3712 core Linux-based HPCC. Four Challenges: 1. The Environment: space, power & cooling 2. How do we manage all of this hardware?! 3. Storage: How do you serve up files to 4000 I/O hungry processes? 4. Cluster Queueing: So you want to queue a million processes? IV. Conclusions: HPC does not scale linearly.

3 What is? One of the most advanced biomedical computing centers in the world. Broad range of computational biomedical and biology research, from biophysics to genomics. More than 15 Labs and roughly 150 faculty, staff, students and postdocs across both campuses and 8 departments. IT staff of 8, covering everything from desktop to HPC & datacenter. Affiliates

4 Before & After Over the past 3 years we have grown our HPC resources by an order of magnitude, driven largely by our Titan compute cluster. Before (2008) After (2011) Total CPU-cores ~500 ~4500 Largest system: CPU 400 core 3712 core Largest system: Memory 800 GB 8 TB Annual CPU-hr 2 M CPU-hrs 25 M CPU-hrs Average Active Users Storage capacity 30 TB ~ 1PB Data Center space 800 sq.ft sq.ft.

5 Titan cluster specs: Titan debut on the Top500.org list as the 124th fastest computer in the world (June 2008) Total CPU/RAM Blade chassis Compute Nodes CPU Memory Primary network Backbone switch InfiniBand Fabric Head nodes Primary storage 3712 cores / 7968 GB 15x HP C x HP BL2x220C + 16x HP BL460C Intel E5450 (8 cores per node) 430x 16 GB/node GB/node 1 GbE to the node / 2x 10 GbE to the chassis Force10 E chassis of Mellanox DDR (20Gbps) IB 2x HP DL360 Isilon Systems, ~1 PB Operating System Fedora Linux 10 Queuing/RM Sun Grid Engine 6.2

6 Titan cluster architecture C2B2 management network /16 CORE network x.x/ x.x/25 Load balancers 4x 1 GbE (Cat6) Isilon 20x 1 GbE (Cat6) Login nodes (XenServer) Titan Cluster Force10 Cluster Router /22 HP /22 HP C7000 Chassis with 32x BL2x220C 10 GbE (MM) 1 GbE (Cat6) Master node HP Blade Chassis with Nodes HP Blade Chassis with Nodes... Shadow master node HP /22 1 GbE (Cat6) Force10 E GbE Switch Provides network backplane CORE network C2B2 management Cluster network Cluster management Three networks: 1. Primary network for MPI communication & Storage 2. Management network for basic services and deployment 3. (not pictured) Chassis management and ilo

7 Challenge 1: The Environment The trouble with ordinary computing environments. A system this size can easily consume between 80 kw and 150 kw of power (not including supporting systems). Typical datacenter power scenarios limit to 5kW - 6kW per rack. Ideally we need to pack these nodes as tightly as possible. If we stuck to a 5kW/rack restriction, we could need between 15 and 30 racks! 3 loaded blade chassis (ideally in 1 rack) consume 20 kw - 30 kw of power. Clearly a standard hosting option wasn t going to cut it.

8 Challenge 1: The Environment Design for an HPC computing environment. We found that an installation this size could no longer depend on resources like power and cooling supplied in typical data centers. What we did: We built out a new, 3000 sq.ft. data center capable of hosting up to 86 racks, designed with HPC in mind. We installed dedicated, in-row cooling systems for dense computing scenarios, enabling up to 25kW per rack. Upgraded building transforms and installed a 1 MW UPS. Installed a 10 GbE central network for the room.

9 Challenge 1: The Environment Titan installed with in-row cooling (above) and hot-aisle containment (below). One (of two) large lead-acid battery banks.

10 Challenge 2: Managing hardware How do you (cleanly) power off 500 servers? Previously we had gone with early blade systems or 1U pizza box clusters with little to no hardware management features. With 464 servers, the enterprise hardware management/deployment becomes a must to efficiently perform simple tasks like powering on/off equipment. Previously we used direct-to-switch networking (i.e. a cable to each host). 464 cables for the primary network cables for the management network = 928 network cables to run? And so on for power management, KVM/serial console access, monitoring...

11 Challenge 2: Managing hardware A real-world example case: 1. By default, the hardware comes configured to PXE boot on NIC But! NIC 1 is not on our management network where installs happen, so we need this option changed to PXE boot from NIC We can use the serial console to access the system BIOS to set this option. i. A quick test run takes approximately 4m30s to go through the boot-up procedure and change the BIOS setting. ii. We have 464 blades. Do the math... that s 35 man-hours of BIOS tweaking! 4. Instead, a good hardware deployment package can be used (in this case, HP s SmartScripting Toolkit). We were able to setup SST an deploy the BIOS change to all of the nodes in < 1 hour!

12 Challenge 2: Managing hardware How blade chassis management saves the day. We can manage power, BIOS, firmware, serial console, etc, all through a unified, scriptable interface. All primary cabling was limited to 15 independent blade chassis in 5 racks, with only 4 network cables each (928 cables down to 60). Physical installation took 3 people about an afternoon of work. And, a perk, by sharing power supplies we keep our power footprint much, much lower. We also had to run only about 1/20th the power cables.

13 Challenge 3: Storage So, you have 4000 processes, and they all want data? Previously we had run simple Linux-based NFS file servers. For our previous largest system (400 cores) we had already had to step up to larger, enterprise NAS devices to get the performance we needed. Even still, it was clear that we weren t going to be able to get the thoughput and IOPS we needed using the standard single-headed NAS approach. Also, we re going to need a lot of disks spinning.

14 Challenge 3: Storage For parallel processes you need parallel file access. It s notoriously difficult to estimate filesystem performance requirements for general purpose HPC, so we need some basic heuristics. A typical stand-alone server would have one reasonably fast disk to 8 cores. So, it seems reasonable that we would want something like one disk per 8-12 cores, or about 300 disks. We don t want processing on these 300 disks backed up through a single entry point, so we want multiple, clustered mount points. This suggest options like: Luster or GFS Multiple independent file servers Clustered NAS

15 Challenge 3: Storage A multi-tiered clustered NAS approach to HPC storage. To clusters InfiniBand Backend A fast pool of storage provides data to cluster with ~300 disks, enhanced with SSDs, and large amounts of CPU and cache. A slow pool of storage provides cheaper archival storage for capacity expansion. Data is automatically migrated to/from here based on access time. A replication storage system provides data protection for large data sets (and is occasionally dumped to tape). 1 Gbps per connection, many connections Isilon node Isilon node Fast storage pool CPU SSD SATA SATA... CPU When old data is touched it comes back here Cache Cache SSD SATA SATA Isilon node Isilon node Replication storage CPU CPU Cache Isilon node Isilon node SATA SATA SATA... Cache SATA SATA SATA... Old data is automigrated here Important data from both pools replicate to the replication storage pool. Slow storage pool CPU SATA SATA SATA... CPU Cache Cache SATA SATA SATA

16 Challenge 4: Queuing You have 1 Million jobs, what runs next? When we had a few hundred (or even thousand) jobs on our systems we could keep a close eye on them to make sure everything was ok. We re going to have 4000 running and 10s or 100s of thousands queued. We need an efficient, fair, reliable queuing system. We did a lot of tweaking to queueing: Robust sharetree + functional queuing for fair (or weighted) queuing. Enforced memory management across the cluster (users must give their own limit, and the queue finds a slot where they fit). Time-bin based queuing to ensure no one can take over the whole cluster and keep it for too long.

17 Challenge 4: Queuing Time-bin based queuing. One of the scenarios we would like to combat is when a single user takes all or most of the slots on the cluster to run very long jobs. Ideally, a user should never have to wait for than the length of their run time (much less really!) for a job to start. We could just limit the number of jobs that one user can submit, but that hurts over-all utilization. CPU 2 12 seconds 2 13 seconds 2 15 seconds Instead, we can limit resources to longer running jobs using overlaid, time-limited queues. We found that jobs tend to divide somewhat evenly into 2 n bins seconds 2 22 seconds... RAM

18 Challenge 4: Queuing Time-bin based queuing, how we determine the bins. 1. We analyzed logs for millions of jobs on older clusters to find typical time characteristics. 2. We found that the runtimes fit a log2 plot fairly linearly, suggesting 2n time bins. 3. Finally, we double check that the time bin sizes maintain somewhat consistent over time. Here we see over-all consistency throughout a year, with some variations in the middle queues. This suggests merging those queues.

19 Conclusion HPC clusters do not scale linearly (but they can scale). To date we have clocked roughly 75 Million compute hours on Titan. We have seen filesystem throughput in excess of 10 Gbps and IOPs well over a million. Despite operating near 100% capacity, users can typically get right in and get their work done. Now we are looking towards the next steps: Shared memory over InfiniBand? GPU acceleration? Virtualized HPC? Thank you! Questions?