Green Cube GSI FAIR Tier 0 Thorsten Kollegger FAIR & GSI 30. March 2015 AIME Big Data - Budapest
Green Cube New data center at GSI, mainly for FAIR Tier0 6 floors, ~25x35 sqm footprint T. Kollegger AIME Big Data, Budapest 30. March 2015 2
GSI & FAIR GSI Helmholtz Center for Heavy Ion Research FAIR Facility for Antiproton and Ion Research Located in Darmstadt, Germany ~1.300 employees and ~3.000 worldwide users T. Kollegger AIME Big Data, Budapest 30. March 2015 3
FAIR Currently under construction Fully operational in the next decade, ~1B EUR investment European project of 8 member states + ~50 international partners T. Kollegger AIME Big Data, Budapest 30. March 2015 4
FAIR Science time Atomic, applied and plasma physics - APPA Hadron structure - PANDA Astrophysics and nuclear structure - NUSTAR QCD phase transition - CBM temperature big bang T. Kollegger 21.11.2014
FAIR Computing The FAIR experiments will produce tens of PetaBytes of raw data per year, with up to 1 TByte/s from the experiments to the Tier0, which need to be safely and permanently stored analyze these datasets over several years, requiring also extensive simulations and theory calculations T. Kollegger AIME Big Data, Budapest 30. March 2015 6
FAIR Computing FAIR Tier 0 @ full FAIR operation 300.000 CPU-cores 35 PByte/year Online-Storage (HDD) 30 Pbyte/year Permanent-Storage (Tape) Comparable resources in FAIR partner countries Key Points Few big, highly efficient data centers Further partner connected via regional federations Co-T0(s) in FAIR Partner Country(-ies) T0@FAIR/GSI T0 Teralink Federation Partners (~T2) T1 s or Regional Federations in FAIR Partner Countries (T1+T2) T. Kollegger AIME Big Data, Budapest 30. March 2015 7
FAIR Tier 0 @ GSI FAIR Tier 0 is the core of the FAIR Computing Model Experiment Online Farms connected via ~4000 fibres to Experiments Experiment Storage Systems Experiment Processing Farm Integration of HPC Centers of surrounding partners (Teralink) Co-T0(s) in FAIR Partner Country(-ies) Distribution of experimental data to FAIR Partners; several 100 Gb/s links T0@FAIR/GSI T0 Teralink Federa on Partners T1 s or Regional Federa ons in FAIR Partner Countries (T1+T2) T. Kollegger AIME Big Data, Budapest 30. March 2015 8
FAIR Data Center A common data center for FAIR (Green Cube) Key Facts 6 floors, 4.645 sqm Space for 768 19 racks (2,2m) 4 MW cooling (baseline) Max cooling power 12 MW Fully redundant (N+1) PUE <1.07 T. Kollegger AIME Big Data, Budapest 30. March 2015 9
An Alarming Truth World wide air-traffic causes around 2 % of the global CO 2 emission just as much as commercial IT! (ca. 30 GW) [Gartner] Volker Lindenstruth (www.compeng.de) May 22, 2012 Copyright, Goethe Uni, Alle Rechte vorbehalten 10
Data Center State of the Art Electrical Power UPS 10% Data Center Cooling 12% Light, etc. 3% IT Equipment 50% PUE = Total Facility Power IT Equipment Power Power Usage Effectiveness Back Cooling 25% PUE = 2 (typical PUE = 1,6 to 3,5) Note: 12C/kWh 1W = 1,05 /a Source: EYP Mission Critical Facilities Inc., New York Volker Lindenstruth (www.compeng.de) May 22, 2012 Copyright, Goethe Uni, Alle Rechte vorbehalten 11
Innovative Cooling System Architecture Cooling Overhead 5% of computer power evaporation cooling tower data center building server room draw-off air air cooling tower fixtures makekupwater equipment container M M Primary Cooling circuit Secondary Works with every commercial computer cooling circuit secondary pump heat exchanger primary pump Volker Lindenstruth (www.compeng.de) 30. März 2015 Copyright, Goethe Uni, alle Rechte vorbehalten
Wet bulb temperature 2010 Volker Lindenstruth (www.compeng.de) 30. März 2015 Copyright, Goethe Uni, alle Rechte vorbehalten 13
Volker Lindenstruth (www.compeng.de) 30. März 2015 Copyright, Goethe Uni, alle Rechte vorbehalten Heiko Ebermann 14
Prizes Green-IT Award Bundesregierung Visionäre Gesamtkonzepte Deutscher Rechenzentrumpreis 2012 energy efficiency Deutscher Rechenzentrumspreis 2013 Visionary Datacenter Architecture Nominated for Deutscher Rechenzentrumspreis 2014 energy efficency DataCenterDynamics EMEA Award 2013 Data Center Blueprint BroadGroup EMEA Awards Special Commendation Energy Efficiency Land der Ideen 2012 for LOEWE-CSC Green Cube Project of the Month, BMBF 5 Nominations with 4 ranks at Data Center Dynamics EMEA Awards 2011, 2012, 2013 2nd rank at Deutsche Internet Awards 2012 Volker Lindenstruth (www.compeng.de) 30. März 2015 Copyright, Goethe Uni, alle Rechte vorbehalten 15
The FAIR Data Center (Green Cube) Space for 768 19 racks (2,2m) 4 MW cooling (baseline) Max cooling power 12 MW Fully redundant (N+1) Can be used for any commercial IT No battery backup required PUE <1.07 Baseline building cost 12 M Volker Lindenstruth (www.compeng.de) 30. März 2015 Copyright, Goethe Uni, alle Rechte vorbehalten 16
Green Cube Construction started in Dec 2014 On schedule for start of operation in Q4/2015 T. Kollegger AIME Big Data, Budapest 30. March 2015 17
Green Cube T. Kollegger AIME Big Data, Budapest 30. March 2015 18
Green Cube Prototype Two floor prototype in Testinghalle @ GSI Home of current GSI Computing Systems T. Kollegger AIME Big Data, Budapest 30. March 2015 19
GSI Scientific Computing Systems Compute clusters Prometheus ~10.000 cores Kronus L-CSC Storage systems ~4.000 cores + HT ~3.000 cores + 4 GPUs/node (AMD S9150/S10000) Hera ~7 PByte netto (Lustre) Nyx ~5 PByte netto (Lustre) x2 by end of year Used for FAIR&GSI Experiments, ALICE Tier2 & National Analysis Facility, Theory T. Kollegger AIME Big Data, Budapest 30. March 2015 20
GSI Scientific Computing Systems Compute clusters Prometheus ~10.000 cores Kronus L-CSC Storage systems ~4.000 cores + HT ~3.000 cores + 4 GPUs/node (AMD S9150/S10000) Hera ~7 PByte netto (Lustre) Nyx ~5 PByte netto (Lustre) x2 by end of year Used for FAIR&GSI Experiments, ALICE Tier2 & National Analysis Facility, Theory #1 in Green 500 list (Nov 2014) 5,27 GFLOPS/W T. Kollegger AIME Big Data, Budapest 30. March 2015 21
TeraLink Tight integration of LOEWE-CSC Computing Center in Frankfurt (~30km away; 84km in fibres) Router cluster (LNET) for the TeraLink 12 x 10 Gbit Ethernet => lustre IB loadbalancing T. Kollegger AIME Big Data, Budapest 30. March 2015 22
TeraLink lustre clients at Gauss Supercomputer Frankfurt Hera lustre cluster at GSI Reading: 13.5 GB/s Writing: 14GB/s Offloading of jobs to Frankfurt, opportunistic resource usage Additional storage system for selected users in Frankfurt T. Kollegger AIME Big Data, Budapest 30. March 2015 23
Software Stack FairRoot Common Software Framework for Simulation, Reconstruction and Analysis for the FAIR Experiments https://github.com/fairrootgroup/fairroot T. Kollegger AIME Big Data, Budapest 30. March 2015 24
Protocol Buffers BOOST ROOT ZeroMQ CMake Geant4 Genat4_VMC Geant3 VGM???? DDS Building configuraion Testing Fair MQ Fair DB MC Application Module Detector Magnetic Field Event Generator Runtime DB FairRoot and ALFA AliRoot6 (O 2 ) CbmRoot PandaRoot R3BRoot SofiaRoot MPDRoot AsyEosRoot FopiRoot EICRoot ALFA FairRoot Libraries and Tools T. Kollegger AIME Big Data, Budapest 30. March 2015 25
Summary Green Cube Data Center at GSI under contruction PUE < 1.07, low Cap. and Op. Exp Supplemented by Software Development and System Design Program for FAIR Computing T. Kollegger AIME Big Data, Budapest 30. March 2015 26