Supercomputer Upgrade (HPC Phase 3) Friday 15 th March 2013

Transcription

1 Supercomputer Upgrade (HPC Phase 3) Friday 15 th March 2013

2 Agenda Item Presented By Time Agenda & Team Introduction Peter Fattoush 5mins Overview Neville Smith 5mins SDC Update Barry Nugent 5mins Oracle Presentation Andrew McArthur / Para Ekanayake 20mins Slash App Slides Robin Bowen 5mins Q&A All 20mins Close Barry Nugent

3 Team Structure Oracle Project Board BoM Project Board Victor Nosiara Ann Farrant Marco de Jonge Gavin Clarke Barry Nugent Rob Lovery Neville Smith ACS Delivery Paul Munro Customer Care Lynn Davis Change Authority & Project Support Debbie Stewart Project Manager Andrew McArthur Interaction with BoM Interaction with ORACLE Project Manager Peter Fattoush Project Support Richard Oxbrow Charity Mackinnon Project Team Jan Harvey Mark Cram Jean Luc Frederic Andrew Scott Project Team James Mandalis Greg Altimore Arnold Mavromatis Robin Bowen

4 Overview Neville Smith

5 Second Data Centre (SDC) Update Barry Nugent

6 SDC Update The SDC will provide redundancy to Bureau operational IT support, so that in the event of an incident partially or fully disabling the Head Office Central Computing Facility (CCF), operational processes will be maintained in the SDC. The Bureau's previous Disaster Recovery Site (DRS) in Canberra had only limited facilities, and could not support the Bureau's operations in the event of an outage involving the CCF. The new facility aligns with the fourth objective of the Bureau Strategic Plan To modernise and extend the capability of our information and communication [infrastructure]. It positions the Bureau to develop a highly robust operational IT infrastructure, while providing a highly stable environment to house the new 140 Teraflop supercomputing systems and support infrastructure. The additional increase in cost allows for five times the facilities capacity from 85kW at the current facility (CDC) to 518kW at the SDC, this capacity is provided at a 50% reduction of the current cost for each kw and allows for immediate capacity available for mid-life HPC Upgrade and to meet the bureau's immediate compute needs.

7 SDC Update

8 Oracle Project Update Andrew McArthur

9 Oracle Project Scope Overview HPC system architecture and design documentation Hardware assembly and installation at site Installation of software Hardware testing of all components Perform acceptance testing Run benchmarks and record Solar II performance Provide training to cover changes to technology or functionality that is not contained in Solar

10 Oracle Project Roles and Responsibilities Document hardware design Document software design Oracle Install and assemble all racks and blades Install IB and Ethernet cabling within racks Install IB and Ethernet cabling between racks Implement provisioning system for infrastructure/ compute Install Applications as supported in existing Solar system Test environment to validate functionality, reliability and availability according to acceptance test plan Project management Acceptance Testing approach Risk and Issue approach Change Management Complete data centre design Bureau of Meteorology Ensure data centre layout to approved design Ensure data centre cooling to meet requirements Technical expertise to support timely decision making Installation of ILOM networking equipment and Ethernet cabling Provide temporary storage at data centre for receiving and install hardware Provide remote and onsite access to the data center as required Ensure power and cable tray design to meet power and cabling requirements Provide sign off according to agreed Quality Management approach

11 Oracle Project Timeline Implementation Timeline Month Feb Mar Apr May Jun Post Acceptance Test (dates to be confirmed) Planning Project Planning Monitor, Control, Report Design Hardware Design Build/Install Software Design Hardware Install Software Install Burn In Acceptance Test Accept Test Operational Readiness Application Porting Parallel Test

12 Oracle HPC Architecture Para Ekanayake

13 HPC Architecture Design HPC Hardware Components Compute Nodes Networking Storage Design Considerations High Level Architecture HPC Software Stack

14 HPC Architecture Design Compute and Infrastructure Nodes Compute Nodes (576 Nodes) X3-2B Blade Servers Sandy Bridge 2 CPU, Sandy Bridge 2.5hz, 6 Core, 100GB SSD, 64GB RAM 10GB Ethernet ( For OS level Provisioning and Management) Blade 6000 Enclosure ( 58 Enclosures) Max 10 Blades per Enclosure (Max 30 nodes per rack) IB EM, 10Gig Connectivity, N+N Redundancy Infrastructure Nodes based on X3-2L ( 46 Nodes) 2RU, Sandy Bridge, Dual PSU, Dual - 8 Core 2.2Ghz,32GB RAM 24x Lustre Nodes ( 12 Failover Pairs) 4x Lustre Metadata ( 1 MDS & 1 MGS Pair) 2x Subnet Manager 2x Admin Nodes 4x Log-in Nodes 2x Operator Console 6x Data Movers 2x Oracle Grid Engine

15 HPC Architecture Design IB & Ethernet Networking IB Switch Fabric ( 36 Port Switch 18 Spine & 36 Edge) Non Blocking Distributed Architecture 18x Spine switches distributed between two separate racks for better availability. Ethernet Design Management /Provisioning 10GbE NEM Switch 10GbE w/24 ports 10GbE ToR Switch 72 10GbE ports (16 QSFP + 8 SFP+) 24x ToR 72P (20 for compute & 4 for Comms & Infra Racks) 116x 24 Port Blade NEMs (2 Per BL Enclosure)

16 HPC Architecture Storage Storage M2 Storage ( QTY 27) OSS Lustre Storage 24x 2540M2 each configured with one 2501M2 tray (Total of 24x 600GB 15K drives per system. (576x600GB =345 TB Raw and 214TB Usable) Raid 6 (8+2) and a pair of mirrored disks for logs 2 sets each per system Lustre MDS Storage ( Metadata Storage) 2x 2540M2 with one 2501M2 tray for each system( Total of 48x300GB 15K drives) 48x 300GB 15K = 14.4TB Raw Using Raid 10 Mirrored Approx 7TB usable) Shared storage between OGE & Admin Servers 1x 2540M2 with 12x600GB 15K = 7.2TB Raw Using Raid 5 Approx 5TB Usable

17 HPC Architecture for 576 Nodes Compute X3-2B M3 (Single Node) MGT & Provisioning -Ethernet Fabric Fat tree IB Fabric IB 54 (18 Spine & 36 Edge switches) Lustre & MDS Storage - (2540M2) Infrastructure & MGT Nodes (X3-2L)

18 HPC Architecture Exemplar System Equipment List Rack 1x 42RU Sun Rack IB 1 x 36 port Ethernet 1x ToR 72p Switch Compute 1x C6000 Enclosure + 10x Compute Nodes Infra 1xOSS Pair, 1xMDS, 1xAdmin, 1xSGE, 1xLogin, 1xDM Storage OSS, MDS ( Single system), SGE & Admin

19 Oracle - Data Centre

20 Oracle - Software Note: As we are still in design phase the software is subject to minor change

21 /apps Environment Robin Bowen

22 Slash App Slides /apps on new HPC System - 2 Modules enable easy access to software packages and libraries /apps filesystem provides cross-platform access to these A single "module load" sets user path and environment variables for each package or library and sets all dependencies Same command on all systems using /apps and modules /apps on new HPC System - 2 On the HPC upgrade of Solar Users will get /apps by default in their environment All standard software packages and libraries will be installed under /apps, including compilers Module commands will remain same as on Solar All packages and libraries from Solar will be available but not all versions will be ported New system is using Linux 6.2 so some changes in the background

23 Slash App Slides /apps on new HPC System - 3 What will NOT be on the new system /g/sc/ophome/nmoc_share/local/nmoc_modules /opt/modulefiles Some versions of packages Versions of packages and libraries ported will be default and latest versions only, except for some special requirements. Disclaimer: /apps is a collaborative effort of NMOC and CAWCR staff with assistance of ITB staff. Support is business hours only except for NMOC staff dealing with Operational issues.

24 Thank you Peter Fattoush BoM Project Manager