Lustre on Hyperion. Marc Stearman. April, 2009

Size: px

Start display at page:

Download "Lustre on Hyperion. Marc Stearman. April, 2009"

Merryl Catherine Greene
7 years ago
Views:

1 Lustre on Hyperion Marc tearman April, 2009 This work performed under the auspices of the U.. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA LLNL-PRE

2 Disclaimer This document was prepared as an account of work sponsored by an agency of the United tates government. Neither the United tates government nor Lawrence Livermore National ecurity, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United tates government or Lawrence Livermore National ecurity, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United tates government or Lawrence Livermore National ecurity, LLC, and shall not be used for advertising or product endorsement purposes. 2

for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights.

3 Hyperion Overview Goals and Purpose Design and Description Networks and Lustre File ystems Ease of Management and Flexibility 3

4 Contributions Intel, Dell, upermicro Processors Nodes Racks Integration Qlogic, Cisco, Mellanox IB & Ethernet switches IB & Ethernet HCAs IB Ethernet s DDN, un, LI torage Hardware Redhat Linux Testing ystem Admin Tri-Lab Knowledge Expertise Facilities 4

Ethernet HCAs IB Ethernet s DDN, un, LI torage Hardware Redhat

5 Hyperion: A Large calable Testbed Development and Testing Environment Infiniband Open ource oftware - OFED Lustre Production caling TO software stack development & testbed Evaluation Testbed For New Hardware & oftware Petascale I/O scaling for equoia and beyond Processor, memory, networking, storage, etc. Designed for future technology refresh Virtualization Vehicle for long term vendor and customer partnerships OpenFabrics Alliance Lustre Center of Excellence un 5

scaling for equoia and beyond Processor, memory, networking, storage, etc.

6 Hyperion calable Unit RP (Management node) Contains diskless images for nodes Login Interactive use nodes Good for compiles, launching jobs, examining files, etc. Lustre LNET s 2 Ethernet 2 Infiniband Batch compute nodes: 138 nodes available 6

compiles, launching jobs, examining files, etc.

7 Hyperion Phase 1 12 TF/s calable Unit ( cores 138 Dual ocket Quad Core Compute Nodes (1, Port IBA 4x Uplinks to spine switch 1GbE Management 12x24 = 288 Port (144D144U) Infiniband 4x QsNet Elan3, 100BaseT Control 1 Login/ ervice/ Master 4 Gateway 1.5 GB/s delivered I/O over 2x10GbE Lustre Object torage ystems 10GbE attached 100 TB and 2.5 GB/s GW Hyperion Phase 1 Deployment is an 4 U 48 TF/s Cluster with full IBA 576 Total nodes and 4,608 cores, 12.1 TB/s memory bandwidth, 9.2 TB capacity IBA is expandable to 1,728 IB ports single plane and can double to dual plane Dual socket 2.5 GHz quad-core Intel Harpertown nodes 8 GB from 4 channels FB-DIMM GB/s Nodes utilize PCI-Express generation 2 I/O which provides an upgrade path to IBA 4x QDR 250 kw of power, 70 tons of cooling. torage calable Units (U) from DDN, LI and un yielding >36GB/s and 1.6 PB disk GW GW GW AN (1/10 GbE and IBA) 1 Remote Partition erver with 2x1 GbE MD MD Lustre MetaData 7

5 GB/s GW Hyperion Phase 1 Deployment is an 4 U 48 TF/s Cluster with full IBA 576 Total nodes and 4,608 cores, 12.1 TB/s memory bandwidth, 9.

8 Hyperion Phase TF/s calable Unit ( cores 138 Dual ocket Quad Core Compute Nodes (1, Port IBA 4x Uplinks to spine switch 1GbE Management 12x24 = 288 Port (144D144U) Infiniband 4x QsNet Elan3, 100BaseT Control 1 Login/ ervice/ Master 4 Gateway 1.5 GB/s delivered I/O over 2x10GbE Lustre Object torage ystems 10GbE attached 100 TB and 2.5 GB/s GW Hyperion Phase 2 Deployment is an 4 U 74 TF/s Cluster with full IBA 576 Total nodes and 4,608 cores, 37 TB/s memory bandwidth, 9.2 TB capacity IBA is expandable to 1,728 IB ports single plane and can double to dual plane Dual socket 2.4 GHz quad-core Intel Nehalem nodes 12 GB from 6 channels of GB/s bandwidth Nodes utilize PCI-Express generation 2 I/O which provides an upgrade path to IBA 4x QDR 250 kw of power, 70 tons of cooling. torage calable Units (U) from DDN, LI and un yielding >36GB/s and 1.6 PB disk GW GW GW AN (1/10 GbE and IBA) 1 Remote Partition erver with 2x1 GbE MD MD Lustre MetaData 8

Control 1 Login/ ervice/ Master 4 Gateway nodes @ 1.5 GB/s delivered I/O over 2x10GbE Lustre Object torage ystems 10GbE attached 100 TB and 2.

9 Hyperion Infiniband Core Diagram Coming with Phase Two 288 Port IB witch Port IB witch Port IB witch Port IB witch Port IB witch Port IB witch 6 Two Two Nodes 144 Nodes 144 Nodes 144 Nodes 144 Nodes 144 Nodes 144 Nodes 144 Nodes U1 U2 U3 U4 U 5-8 Coming with Phase Two (Integrating right now) 9

Two 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 144 Nodes 144 Nodes 144 Nodes 144 Nodes 144

10 Hyperion Compute Core Nodes dhcp boot across the management network. The RP nodes contain the diskless images for nodes. Most system traffic (syslog, ntp, etc) travels across the management network along with NF access to home directories. Lustre and MPI traffic for jobs travels across the IB core network and to the Lustre networks via the Lustre routers. Remote power control and consoles have an isolated network. /home /usr/local NF Home Lustre Hyperion IB core MPI network /21 LNET: o2ib0 Compute Login RP Hyperion management ethernet network /20 Hyperion console and power control ethernet network /24 cylades PDUs 10

Lustre and MPI traffic for jobs travels across the IB core network and to the Lustre networks via the Lustre routers.

11 Hyperion File ystems and Networks /p/l_terascala /p/l_sun Terascala un DDN1 /p/l_ddn[1-3] DDN2 LI Lustre IB AN /24 LNET: o2ib1 Lustre Ethernet AN /24 LNET: tcp0 DDN3 /p/l_lsi IB LNET Eth LNET /home /usr/local NF Home Hyperion IB core MPI network /21 LNET: o2ib0 Compute Login RP Hyperion management ethernet network /20 11

0/24 LNET: tcp0 DDN3 /p/l_lsi IB LNET Eth LNET /home /usr/local NF Home Hyperion IB core MPI

12 Multiple Lustre Versions un File system testing lustre 1.8 All other file systems running 1.6.x Clients and routers have been segrated to run 1.6 or 1.8. un and LLNL started Interoperability testing a couple weeks ago. A slurm partition called lustre_18 has been created for compute nodes that mount lustre 1.8. The 1.6 clients are still in the pbatch partition 12

6 or 1.8. un and LLNL started Interoperability testing a couple weeks ago.

13 Flexible Configuration ervers, Clients and s run diskless nfsroot images Can be quickly switched to new software stack with a reboot lurm partitions can be created to group nodes with similar features/software stacks Having six lustre file systems allows us to test many different versions ize of Hyperion allows us to test and reproduce bugs we see in production 13

with similar features/software stacks Having six lustre file systems allows us to test many

14 un Lustre File system 35TB 12TB RAW 12TB RAW MD O1 Lustre IB AN /24 LNET: o2ib1 IB LNET x4 Login Compute RP 12TB RAW O2 IB LNET Hyperion IB core MPI network /21 LNET: o2ib0 12TB RAW O3 Eth LNET 12TB RAW O4 Lustre eth AN /24 LNET: tcp0 10Gb Eth x4 12TB RAW O5 10Gb Eth Eth LNET 14

15 LI Lustre File ystem 284TB Compute 80TB Raw 80TB Raw 80TB Raw LI LI LI LI LI LI O1 O2 O3 O4 O5 O6 Lustre IB AN /24 LNET: o2ib1 IB LNET IB LNET x4 Login Hyperion IB core MPI network /21 LNET: o2ib0 RP 80TB Raw LI LI O7 O8 MD 15

0/24 LNET: o2ib1 IB LNET IB LNET x4 Login Hyperion IB core MPI

16 DDN Lustre Network Diagram 2A 9900 IBA 4x 160x147GB A (19TB) 2A 9550 IBA 4x DR 512x250GB ATA (91TB) 2A 9900 IBA 4x 280x500GB ATA (103TB) Compute MD3 Login RP 120TB Raw DDN 9550 DR O1 O2 10Gb Eth 10Gb Eth Eth LNET 140TB Raw 23.5TB Raw DDN 9900 DDN 9900 MD2 O1 O2 O1 O2 MD1 10Gb Eth Lustre eth AN /24 LNET: tcp0 x4 Eth LNET Hyperion IB core MPI network /21 LNET: o2ib0 16 bay A JBOD 16

Eth Eth LNET 140TB Raw 23.5TB Raw DDN 9900 DDN 9900 MD2 O1 O2 O1 O2 MD1 10Gb Eth Lustre eth AN 172.16.

17 Terascala Lustre File ystem 14TB RT1000 MD O1 O2 O3 DR Lustre IB AN /24 LNET: o2ib1 IB LNET x4 Hyperion IB core MPI network /21 LNET: o2ib0 Compute Login O4 IB LNET RP 17

0/24 LNET: o2ib1 IB LNET x4 Hyperion IB core MPI

18 Discussion 18

SOLUTION BRIEF AUGUST 2015. All-Flash Server-Side Storage for Oracle Real Application Clusters (RAC) on Oracle Linux

SOLUTION BRIEF AUGUST 2015. All-Flash Server-Side Storage for Oracle Real Application Clusters (RAC) on Oracle Linux AUGUT 2015 All-Flash erver-ide torage for Oracle Real Application Clusters (RAC) on Oracle Linux Introduction Traditional AN storage systems cannot keep up with growing application performance needs. The