Transforming the UL into a Big Data University. Current status and planned evolutions

Transcription

1 Transforming the UL into a Big Data University Current status and planned evolutions December 6th, 2013 UniGR Workshop - Big Data Sébastien Varrette, PhD Prof. Pascal Bouvry Prof. Volker Müller

2 Preamble Classical Storage metrics Storage capacity: multiple of bytes (TB: TiB= ) Transfer rate on a medium: Mb/s or MB/s Other metrics: Sequential vs Random R/W speed, IOPS 2

3 Preamble Classical Storage metrics Storage capacity: multiple of bytes (TB: TiB= ) Transfer rate on a medium: Mb/s or MB/s Other metrics: Sequential vs Random R/W speed, IOPS The Big Data Challenge : 4 V s [ Volume Velocity Variety Veracity ] Also relevant for Luxembourg s research priorities Large number of diverse data sources to integrate Not just about storage capacity! 2

4 Preamble Classical Storage metrics Storage capacity: multiple of bytes (TB: TiB= ) Transfer rate on a medium: Mb/s or MB/s Other metrics: Sequential vs Random R/W speed, IOPS The Big Data Challenge : 4 V s [ Volume Velocity Variety Veracity ] Also relevant for Luxembourg s research priorities Large number of diverse data sources to integrate Not just about storage capacity! In this talk: Storage UL 2

5 Overview of Storage Components

6 Storage Levels Larger, slower and cheaper CPU Registers L1 - C a c h e L2 - C a c h e L3 - C a c h e Memory Bus Memory I/O Bus register reference L1-cache (SRAM) reference L2-cache (SRAM) reference L3-cache (DRAM) reference Memory (DRAM) reference Disk memory reference Level: Size: Speed: 500 bytes 64 KB to 8 MB 1 GB 1 TB sub ns 1-2 cycles 10 cycles 20 cycles hundreds cycles ten of thousands cycles HDD: 7,2 krpm) R/W: 100 MB/s; 190 IOps SDD: R/W: 560 MB/s; IOps 4

7 HDD vs. SSD Performances HDD: 150 SDD:

8 Interconnect Overview of the Main HPC Components Latency HPC Components: Interconnect time to send a minimal (0 byte) message from A to B latency: Bandwidth time to send a minimal (0 byte) message from A to B max amount of data communicated per unit of time bandwidth: max amount of data communicated per unit of time Technology E ective Bandwidth Latency Gigabit Ethernet 1Gb/s 125 MB/s 40µs to300µs Myrinet (Myri-10G) 9.6 Gb/s 1.2 GB/s 2.3µs 10 Gigabit Ethernet 10 Gb/s 1.25 GB/s 4µs to5µs Infiniband QDR 40 Gb/s 5GB/s 1.29µs to2.6µs SGI NUMAlink 60 Gb/s 7.5 GB/s 1µs 6

9 Data Management Storage architectural classes & I/O layers Application [Distributed] File system SATA SAS FC... iscsi... Network NFS CIFS AFP... Network DAS Interface SAN Interface NAS Interface SATA DAS Fiber Channel Ethernet/ Network Fiber Channel Ethernet/ Network SAS Fiber Channel SATA SAN File System NAS SAS SATA Fiber Channel SAS Fiber Channel 7

10 Data Management / HW Protection RAID standard levels RAID combined levels 8

11 Data Management / File System (FS) Logical manner to store, manipulate and access data Disk file systems FAT32, NTFS, HFS, ext3, ext4, xfs... 9

12 Data Management / File System (FS) Logical manner to store, manipulate and access data Disk file systems FAT32, NTFS, HFS, ext3, ext4, xfs... Network file systems NFS, SMB 9

13 Data Management / File System (FS) Logical manner to store, manipulate and access data Disk file systems FAT32, NTFS, HFS, ext3, ext4, xfs... Network file systems NFS, SMB Distributed and/or Parallel file systems data are stripped over multiple servers for high performance generally add robust failover and recovery mechanisms Lustre, GPFS, FhGFS, GlusterFS GFS (Google), HDFS (Hadoop), OneFS (Isilon) etc. 9

14 Storage HW Components Hosting High density disk enclosures includes [redundant] HW RAID controllers RAID Controller card performances differs! Basic (low cost): 300 MB/s; Advanced (expansive): 1,5 GB/s Typical encl. sizing: 4U / 48 to 60 disks of 4TB 10

15 Storage HW Components Hosting High density disk enclosures includes [redundant] HW RAID controllers RAID Controller card performances differs! Basic (low cost): 300 MB/s; Advanced (expansive): 1,5 GB/s Typical encl. sizing: 4U / 48 to 60 disks of 4TB Storage racks: 42U capacity, 15 kw HPC rack: kw Interconnect rack: 6-8 kw 10

16 Storage HW Components Hosting High density disk enclosures includes [redundant] HW RAID controllers RAID Controller card performances differs! Basic (low cost): 300 MB/s; Advanced (expansive): 1,5 GB/s Typical encl. sizing: 4U / 48 to 60 disks of 4TB Storage racks: 42U capacity, 15 kw HPC rack: kw Interconnect rack: 6-8 kw Server rooms Power (UPS, battery), Cooling, Fire protection... 10

17 Data UL : Overview

18 Data UL - SIU Central storage service operated by the SIU for > 5Y on central file server - capacity 30TB including backup / archiving for all university users archived data versions takes about 180 TB (uncompressed). originally developed to secure important user data Current Situation (Dec. 2013) Central Administration: 3 TB User Data: 20 TB Research Data : 196 TB 12

19 Data UL - SIU Central storage service operated by the SIU for > 5Y on central file server - capacity 30TB including backup / archiving for all university users archived data versions takes about 180 TB (uncompressed). originally developed to secure important user data Current Situation (Dec. 2013) Central Administration: 3 TB User Data: 20 TB Research Data : 196 TB undersized for now 2Y 12

20 Data UL - SIU Central storage service operated by the SIU for > 5Y on central file server - capacity 30TB including backup / archiving for all university users archived data versions takes about 180 TB (uncompressed). originally developed to secure important user data Current Situation (Dec. 2013) Central Administration: 3 TB? User Data: 20 TB Research Data : 196 TB undersized for now 2Y 12

21 Data UL - HPC 13

22 UL HPC Platform 2 geographical sites 3 server rooms 3 admins (+ 1.5 in 2014) 4 clusters: 387 nodes, 4110 cores (43.21 TFlops) TB (raw storage, incl. backup) NFS + Lustre > 5.7 M Hardware investment so far Open-Source software stack Debian, SSH, OpenLDAP, Puppet, FAI... 14

23 UL HPC Platform 15

24 Example: the gaia Cluster Uni.lu (Belval) Kirchberg (chaos cluster) Gaia cluster Gaia cluster characteristics Uni.lu - Computing: 250 nodes, 2408 cores; Rpeak 21,62 TFlops 10 GbE - Storage: 240 TB (NFS) + 576TB (NFS backup) TB (Lustre) 10 GbE 1 GbE Gaia cluster access Bull R423 (2U) Cisco Nexus C GbE (2*4c Intel Xeon 2,26 GHz), RAM: 16GB 10 GbE Adminfront LCSB Belval Computing nodes 1x BullX BCS enclosure (6U) 4 BullX S6030 [160 cores] Bull R423 (2U) (2*4c Intel Xeon 2,26 GHz), RAM: 16GB Columbus server (16*10c Intel Xeon E7-4850@2GHz), RAM: 1TB Infiniband QDR 40 Gb/s (Fat tree) IB 2x Viridis enclosure (4U) 96 ultra low-power SoC [384 cores] IB Bull R423 (2U) (1*4c ARM Cortex A9@1.1GHz), RAM: 4GB (2*4c Intel Xeon 2,26 GHz), RAM: 16GB 1x Dell R820 (4U) IB MDS1 NFS server (2*6c Intel Xeon L5640@2.26GHz), RAM: 24GB Bull R423 (2U) (2*4c Intel Xeon L5630@2,13 GHz), RAM: 24GB FC8 FC8 FC8 Bull R423 (2U) 5x Bullx B enclosure (35U) 60 BullX B500 [720 cores] GPU cores FC8 20 disks (600 GB SAS 15krpm) Multipathing over 2 controllers (Cache mirroring) 2 RAID1 LUNs (10 disks) 6 TB (lvm + lustre) FC8 10 GbE FC8 (2*4c Intel Xeon L5630@2,13 GHz), RAM: 96GB Lustre Storage Bull R423 (2U) Nexsan E60 (4U, 12 TB) [32 cores] (4*8c Intel Xeon E5-4640@2.4GHz), RAM: 1TB 12 BullX B506 [144 cores] (2*6c Intel Xeon L5640@2.26GHz), RAM: 24GB 20 GPGPU Accelerator [12032 GPU cores] 4 Nvidia Tesla M2070 [448c] 20 Nvidia Tesla M2090 [512c] (2*4c Intel Xeon L5630@2,13 GHz), RAM: 96GB Nexsan E60 + E60X (240 TB) MDS2 OSS2 Bull R423 (2U) (2*4c Intel Xeon L5630@2,13 GHz), RAM: 48GB FC8 Bull R423 (2U) (2*4c Intel Xeon L5630@2,13 GHz), RAM: 48GB GPU cores FC8 FC8 OSS1 120 disks (2 TB SATA 7.2krpm) = 240 TB (raw) Multipathing over 2+2 controllers (Cache mirroring) 12 RAID6 LUNs (8+2 disks) = 192 TB (lvm + xfs) GPU cores 2*Nexsan E60 (2*4U, 2*120 TB) 2*60 disks (2 TB SATA 7.2krpm) = 240 TB (raw) 2*Multipathing over 2 controllers (Cache mirroring) 2*6 RAID6 LUNs (8+2 disks) = 2*96 TB (lvm + lustre) 16

25 UL HPC: HW Investments 2500#000# # UL#HPC:#Total#Yearly#HW#Investment#(VAT#including)# Other# 2249#294# # Power#Supply# 2000#000# # So:ware# Interconnect# Servers# 1500#000# # Storage# CompuDng#Nodes# 1000#000# # Server#room(s)#/#Racks# 1039#410# # 980#834# # 828#178# # 500#000# # 413#482# # #0# # 25#772# # 119#274# # 93#187# # 2006# 2007# 2008# 2009# 2010# 2011# 2012# 2013# Cumulative Investment:

26 UL HPC: HW Investments (VAT incl.)* 1200!000,00!! UL#HPC:#Total#HW#Investment#(VAT#incl.)#excl.#server#rooms# Other! 1000!000,00!! Power!Supply! So:ware! 800!000,00!! 600!000,00!! Interconnect! Servers! Storage! CompuDng!Nodes! 400!000,00!! 200!000,00!!!0,00!! 2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013! * excluding server rooms 18

27 UL HPC: Power and cooling UL%HPC:%Power%&%Cooling%Usage%in%UL%server%rooms% 250# Max#Availaible#Power#Capacity#[kW]# 229# Server%Room%Capacity%[KW]% 200# 150# 100# Max#Used#Capacity#[kW]# 89,35# 130,63# 179,41# 50# 0# 42,44# 27,67# 29,17# 12,18# 0,45# 2006# 2007# 2008# 2009# 2010# 2011# 2012# 2013# 19

28 UL HPC Storage UL$HPC$Storage$[TB]$ 1400" 1200" 1052,4" 1208,4" Raw$Capacity$[TB]$ 1000" 800" 600" 400" 200" 0" 576" 512,4" 220" 0" 6,6" 6,6" 6,6" 32,4" 2006" 2007" 2008" 2009" 2010" 2011" 2012" 2013" Nyx" G5K" Gaia" Chaos" TOTAL" Backup" 20

29 UL HPC Storage: 1208 TB (raw.) in 2013 NFS-based storage home / work directories directories Lustre-based storage SCRATCH Backup devices mainly for project specific data [392 TB] [240 TB] [576 TB] 21

30 UL HPC Storage Benchmarking IOZone based - increasing #nodes Lustre: NFS (read) I/O bandwidth (GB/s) I/O bandwidth (MiB/s) Lustre, write, filesize 20G Lustre, read, filesize 20G NFS, read, filesize 20G

31 UL HPC and HPCthe in Grande the Grande Region region and Around TFlops TB FTEs Country Name/Institute #Cores R peak Storage Manpower Luxembourg France Germany Belgium UK UL CRP GL TGCC Curie, CEA n/a LORIA, Nancy ROMEO, UCR, Reims Juqueen, Juelich n/a MPI, RZG n/a 5 URZ, (bwgrid),heidelberg UGent, VCS n/a CECI, UMons/UCL > 4 Darwin, Cambridge Univ n/a Legion, UCLondon Spain MareNostrum, BCS S. Varrette, PhD. (UL) HPC UL N 23 3

32 Big UL: planned evolutions

33 Cloud Storage Analysis Study initiated by LCSB (Credits: Bob Pepin) Transfer rate / time should not be under-estimated Requirement [TB] Yearly Cost [K ] Total Cost [K ] , ,006 Legal aspects of data externalization on Cloud? 25

34 UL Milestones Early 2014: RFP for the acquisition of Big UL NAS Scalable NAS with initial effective capacity > 1PB Storage / backups of all devices (user + research) High-performant interface with HPC and UL IT mounting on computing nodes via IB QDR interconnect mounting on desktop / workstation via [new] UL network Continuous increase of HPC storage capacity SCRATCH (Lustre based) - 100K / + 180TB / + 3 GB/s 26

35 UL Milestones UL moves to Belval! 2016: CDC (Centre de calcul) in Belval 3 storage server rooms (68 15 kw) 2 HPC server rooms (52 40kW) Note: 1 floor in addition for SIU + Partners New Belval Campus 27

36 2014 UL Milestones Phase 1 of UL internal cloud infrastructure 28

37 Conclusion / UL: A Necessity Required to correctly handle UL especially on life-science topics Access/Synchronization tool Backup/Archiving tool Expected Budget asked (part of next 4Y Plan) 2014: BIG UL NAS [1 M ] 2015: Medium size tape library / Cloud connector [1.6 M ] 2016: Capacity ext. (Big UL NAS) by 1PB [800 K ] 2017: Capacity ext. (Tape / Archiving) [700 K ] 2018: NAS Duplication to physically separated site [1.5 M ] 29