Huygens Architecture. Jeroen Engelberts - Consultant SARA Computing and Networking Services Science Park 140 Amsterdam

Transcription

1 Jeroen Engelberts - Consultant jeroene@sara.nl Science Park 140 Amsterdam

2 Overview General Architecture Endianness Processors and Memory Affinity Simultaneous Multi-Threading (SMT) Internode networking File systems Questions & Answers

3 General Architecture > Power6 processor: dual-core 4.7GHz (18 GFlops per core) > Total 108 nodes SCN s (standard compute nodes) en MCN s (memory compute nodes) with each: 16 dual-core processors GB shared memory (4-8GB/core) 2 quad-port Infiniband adapters 1 Front-end nodes > > > > > > Total nr cores: 3456 Total memory: 15,75 TB Interconnect: 8 IB switches, non-blocking SAN disk storage: 972 TB raw, 711 TB usable Energy usage: 715 kw Operating system: SUSE Linux Version 11 - SP1

4 Endianness (1) Origin: Gulliver s Travels, dispute between Lilliput (GB) and Blefuscu (F) regarding the opening of eggs introduced in ICT by Danny Cohen in 1980 How the hexadecimal value 0x89ABCDEF is stored:

5 Endianness (2) Huygens Big Endian PCs (Intel, AMD) Little Endian Transport of binary data files not trivial per se Options Text files instead of binary data files. NetCDF (Network Common Data Form) HDF (Hierarchical Data Format) XLFRTEOPTS=ufmt_littleendian=unit list (runtime)

6 Processors and Memory (1) NUMA (Non uniform memory architecture) effects occur in Huygens due to design. Each node consists of four quads, or multi-chip module (MCM). Each quad has four processors. Each processor is a dual chip module (DCM). Three levels of cache memory L1 (64kB instruction, 64kB data) on core L2 (4 MB) on chip L3 (32 MB) on MCM

7 Processors and Memory (2) POWER6 chip 32 MB L3 8B x2 rd 8B x2 wr 4B/8B x2 Core Ghz Core Ghz 128kB L1 128kb L1 4 MB Private L2 4 MB Private L2 L3 Control Directory I/O Ctlr 4B read 4B write Fabric Switch Memory Ctlr I/O Interface 4B/8B x2 Off-Node Fabric 8B x2 read 4B x2 write 2B/8B x2 2B/8B x2 2B/8B x2 On-Node Fabric Memory

8 Affinity It is possible to assign tasks to specific cores manually. On Huygens, affinitization is automatically done by LoadLeveler submit filter from SARA. More details on affinity later today.

9 Simultaneous Multi-Threading (1) Each node contains 32 cores. With SMT each core is split into two virtual CPUs, leading to 64 CPUs. Idea make use idle CPU time: Fetching data costs time, normally core has to wait With SMT core, or other virtual CPU, can perform instructions during this time. User programs may, or may not, benefit from it. SARA nodes have SMT on by default.

10 Simultaneous Multi-Threading (2) SMT can be controlled with specific LoadLeveler keywords: in this way you can execute up to 64 MPI tasks per node. If you want to run in Single Thread (ST) mode instead, you have to specify:

11 Internode Networking Huygens equipped with Infiniband network, each node has 2 Infiniband adapters. Each adapter has 4 ports. The 8 ports cabled to 8 separate switches. Port has throughput of 20 Gbit/s, aggregated to 160 Gbit/s. This comes down to 2 GB/s, aggregated to 16 GB/s.

12 File systems (1) On Huygens, GPFS (General Parallel File System) is used. Advantages of GPFS Load sharing Fault redundancy Huygens does not have a layer of dedicated network storage devices (NSDs) Huygens has an all to all setup, all nodes are connected to all storage boxes all nodes serve as NSDs

13 File systems (2)

14 File systems (3) /home home directory of user Quotum (default) 200 GB / user Being backed up Divided in 9 parts of 14 TB / each Aggregated speed 540 MB/s One such part can be restored in a day from tape /scratch scratch directories of user Quotum 8 Tb on /scratch/local and 8 Tb on /scratch/shared Not being backed up /scratch/shared available to all nodes /scratch/local available to each node separately /scratch/shared aggregated speed of 26.4 GB/s /scratch/local aggregated speed of 21 GB/s Retention time: at least two weeks (currently)

15 File systems (4) /project project space Special to be requested for projects Not being backed up Divided in 2 parts, one slow one fast Capacity/Aggregated speed Slow: 132 TB 5.3 GB/s Fast: 211 TB 21 GB/s /archive Directory for archive purposes Disk cache with tape robot Older files take long to retrieve Virtually no limit tapes can be added Preferably store large files tar, zip

16 Questions and Answers?