Site Presenta+on PSNC

Transcription

1 Site Presenta+on PSNC Norbert Meyer R. Mikołajczak, A. Kaszuba, M. Pospieszny, R. Januszewski

2 Environment: 300+ sqm of data center air, liquid coolled systems video monitoring, fire protec+on 24h monitoring Part of: European HPC infrastructure (PRACE) European and na+onal grid infrastructure (EGI, PL- GRID) Na+onal (Platon, NDS) and European data infrastructure (EUDAT) HPC Center Total peak performance 300+ TFlops, 15K CPU cores, disk arrays 1+ PB, storage capacity ~4 PB 4xTOP500

3 HPC infrastructure PC clusters GP- GPU clusters Prototypes on future promising technologies DATA infrastructure Hierarchical data infrastructure Fast external source of data (compu+ng) Part of Na+onal Data Storage Part of European Data Infrastructure

4 LimitaAon of the current DC Limited space divided into 2 rooms: networking (60 sqm) and real data centre (300 sqm) Cooling Two technologies: air cooled and liquid cooled systems in the same room Problem with cooling capacity (more capacity means more electricity) Roof: limited resistance (heavy cooling condensers) Electricity Currently max capacity - electrical transformer 1,1 MVA Everything above means a necessity to upgrade the transformer Loca+on City cente Inside of a building of Polish Academy of Sciences No chance to upgrade the space Close to a library and hotel - places where silence is an advantage

5 Container based soluaon

6 Container data centre To build a data centre in a new place A temporary solu+on container data centre 120 sqm The final data centre 1370 sqm

7 Container data centre - parameters 4 rows x 5 servers Liquid cooling for each rack 30 kw by rack power consump+on Free Cooling Chiller: Cooling capacity: 400 kw 100% Free cooling: + 3,9 oc Power supply 1 MVA 500 kw for container (servers and cooling system) Reuse of mainly all parts in the final data centre chiller 1 MVA transformer inrows

8 Container - inside Sample arrangement computer room mee+ng room, opera+on stockroom

9

10 8 basic boxes make the 120 sqm (12 m x 10 m)

11

12 cables mounted on the ceiling

13 Building plot 3. phase of building 2. phase of building Data Centre

14 Costs (incl. VAT) 8 containers: total price 25k Euro Power supply: 50k Euro Cooling chiller and power distribu+on 200 k Euro total 275 keuro Lesson learned Scalable architecture Very fast process of building few months Cheap

15 PSNC Data Center (under construcaon)

16 2 Phase Offices + laboratories 1 Phase Data Centre

17 Close to the Warta river Close to Technical University

18 Data Centre 3 levels basement for infrastracture (power supply, cooling, generators) 2 rooms for servers: 2 x 812,9 m^2 roof for liquid chillers Total area of the building: 3731 sqm Net area of compu+ng rooms: 1626 sqm Height: 17,5 m

19 Level height 6,5 m HPC floor Internet services floor low power consump+on

20 Max 208 racks on each floor Max floor load kg/sqm

21 LocaAon Placed near Warta river located above 300 year flood water level Ground area: sqm Area for science and higher educa+on Close to Poznań University of Technology

22 Scalability up to 208 rack on each floor 2 x 13 rows x 8 racks from 1 to 8 transformers (2,5 MVA 20 MVA) 15 MVA for servers with one PSU 2 x 2 MVA emergency diesel generator used as a backup power supply up to 14 MW liquid cooling placed on the roof 2 MW precision air cooling on each floor 12 MW direct server cooling for HPC floor

23 Flexibility Step by step power supply and air condi+on growth Possibility to connect power or cooling system to: tree genera+on system river cooling system any other innova+on system Reuse of energy to heat offices or water Free area for new infrastructure building

24 Very Early Smoke Detec+on System (eg. VESDA) High- Pressure Water Fog system - fire protec+on (e.g. Hi- Fog) Classical fire protec+on system Security Access Control System CCTV video monitoring

25 Power distribuaon system

26

27

28

29

30

31

32

33

34

35 Power supply 2 independent power supplies B MV sta+ons A, B Data Centre A

36 Reference to PRACE White Paper Recommenda+on 1: Provide only the necessary level of redundancy Only cri+cal systems powered by fully redundant power supply Recommenda+on 2: Implement redundancy in a clever way : Two power supply lines, scalable UPS architecture : kw units in parallel Scalable UPS and diesel generators

37 Reference to PRACE White Paper Recommenda+on 3: Internal experts Recommenda+on 10: Consider regula+on and safety : Requirements for trainings is a part of every public tender Separated rooms for MV and LV switchboards Full Access control for technical floor

38 Reference to PRACE White Paper Recommenda+on 4: Think TCO Scalable infratructure that grows on demand Fine- grained extensions ( 0,4/1/2,5 MVA) Simple structure = easy maintenance Recommenda+on 5: Reduce electricity conversion losses : Modular UPS only for cri+cal systems HPC/high power system without UPS Recommenda+on 7: Monitor the power distribu+on system : Monitoring devices in every LV switchboard Central data aquisi+on system Thermal Imaging Camera already in toolbox Recommenda+on 8: Minimize the impact of maintenance : two independent power supply lines and two independent power rails Procedures To Be Done Recommenda.on 9: Select the right electricity provider It doesn t exist a free market in Poland

39 Network ConnecAons Connec+on between the main and backup data centre Connec+on of data centre with PIONIER and GEANT

40 ConnecAng two data centres DC 1 (old) DC 2 (new) 100Net

41 100Net network topology Koszalin Olsztyn Toruń Zielona Góra Szczecin Gdańsk KDM Białystok Puławy Bydgoszcz Poznań KDM Warszawa NASK/KDM Radom Łódź Lublin 100Gb/s Wrocław KDM Kraków KDM Kielce N x 100Gb/s Opole Śląsk Częstoc howa Rzeszów

42 100Net HPC access point

43 100Net facts A new genera+on DWDM transmission network with support of 100 Gbps 100 Gbps network cards LLE and Infiniband switches in each HPC access point (low latency switches) Network for na+onal and interna+onal infrastructure projects (PRACE, EGI, EUDAT, Na+onal Data Storage, cloud services)

44 Reference to PRACE White Papers Selec+ng a Loca+on for an HPC Data Centre Electricity in HPC Centres Cooling making efficient choices Publicly available at: hqp:// ri.eu/

45 Advantages New buidling and infrastructure designed for the DC purposes vs. old building we currently have Placed by the Warta river Experiments done with PRACE prototypes a direct return back investment! Redundancy, scalability and reliability included in the design phase High importance put on low maintenance cost

46 LimitaAons and boundaries Limited size of the building plot It could be 1 level DC instead of 3 floors Short +me period to close the investment (4Q2014) Public tender procedures Less flexibility during the whole process geologically weak base increased the star+ng costs Historical place may delay the process Cutng down trees is a very expensive process Limited height of the DC building

47 Thank you very much!!

48