Beyond High Performance Computing: What Matters to CERN

Size: px

Start display at page:

Download "Beyond High Performance Computing: What Matters to CERN"

Alexina Palmer
7 years ago
Views:

1 Beyond High Performance Computing: What Matters to CERN Pierre VANDE VYVRE for the ALICE Collaboration ALICE Data Acquisition Project Leader CERN, Geneva, Switzerland

2 2 CERN CERN is the world's largest particle physics centre funded in 1954 by 20 European member states Particle physics is about: elementary particles of which all matter in the universe is made fundamental forces which hold matter together Particles physics requires: special tools to create new particles: the accelerators special instruments to study new particles: the experiments P. Vande Vyvre - CERN

3 3 What s Next? Physics of 21 st century will explore unresolved physics questions: What's the origin of the mass of particles? Search for Higg s boson Can all the forces be unified? Grand Unification Theory (GUT) "Dark matter and energy" (96 % of universe) is not visible What is it made of? Search for new particles forming the dark matter Where did the antimatter go? Explore asymmetry between matter and antimatter Explore the matter deeper and deeper using a new accelerator: the LHC

4 4 Large Hadron Collider (LHC) LHC 27 km circumference 100 meters underground Geneva airport

5 5 Inside the LHC

6 6 The Experiments: Underground Cathedrals Computing Centre

7 7 ALICE Experiment at LHC A brief history of ALICE : Design : R&D : Construction : Installation : Commissioning : Operation : Upgrade Detector: 18 technologies Size: 16 x 26 meters Weight: 10,000 tons Collaboration: > 1000 Members > 100 Institutes > 30 countries

8 8 Data Deluge 100 million measurement channels 40 million potential collisions/second

9 9 Data Selection Multi-level trigger system (40 MHz a few khz) Reject background Select most interesting collisions Reduce total data volume

10 10 Data Acquisition Data Acquisition: Acquire data from a matrix made of millions of channel and pertaining to the same physics event (particle collision)

11 11 Data Quality Monitoring Selection: no undo possible Data Quality Monitoring: fast verification of the event selection

12 12 Trigger - Data Acquisition - Offline Trigger (decision) + Data Acquisition (action) Processing Online Offline Collision Detectors Event Fragments Complete Event Data Storage Reconstruction & Analysis Drawing by S. Chapeland

13 13 The Mission of Trigger-Data Acquisition in a Physics Experiments Particles Data Inspect data resulting from all collisions Select the most interesting ones Store them and make them available for off-line analysis Data is our most precious resource! Information Knowledge

14 14 Data Produced at CERN Recording (Mass Storage) Data Archived (Events/s) (MB/s) Total/Yr (PB) ALICE Pb-Pb ATLAS pp CMS pp LHCb pp

15 15 Data Volume from Experiments For each of the 4 experiments: collisions of interest per second 1-50 MB of digitised information for each collision days of data taking per year Recording rate of GB/sec 1-6 Petabyte/year/experiment For the whole of CERN: ~ 15 PB/year during 15 years

16 16 Challenge 1: Data Throughput The overall move of LHC data corresponds to 1 DVD per second during the whole year

17 17 Challenge 2: Data Volume The LHC data corresponds to 1 pile of CDs as high as the Mont-Blanc (4800 m) every year for each experiment

18 18 Challenge 3: Data Access 15 PB/yr used by 8000 physicists from 580 institutes in 85 countries The LHC Computing Grid (LCG): Share the data and pool the resources The world's largest international scientific Grid.

19 19 ALICE DAQ Architecture Event Fragment Rare/All BUSY CTP LTU TTC FERO FERO L0, L1a, L2 L0, L1a, L2 120 DDLs 430 D-RORC 360 DDLs BUSY LTU TTC FERO FERO Load Bal. LDC 125 Detector LDC LDC LDC 10 HLT LDC LDC LDC DDL H-RORC 10 DDLs 10 D-RORC FEP FEP HLT Farm Sub-event EDM Event Building Network Event File GDC TDS GDC 75 GDC 30 TDSM TDSM Storage Network 75 TDS TDS TDSM DA DQM 60 DA/DQM ACR 30 ACR Archiving on Tape in the Computing Centre (Meyrin) DSS 18 DSS PDS

20 Storage System: Key Design Factors Large project with a long duration: long term support, evolution and interoperability are key design factors First design of the Data Acquisition System in 1995 Together with the budget request! R&D and tests: - SNW conference: technology in action, contacts with vendors - Validation in our lab of the Fibre Channel technology (FC2G) - Selection of hw components. Loan of components is essential! Architecture coherence Selection of the cluster file system Staged Deployment and commissioning - Adoption of FC4G - Deployment in several stages (10, 20, 40, 100 % of the performance). The SAN has nicely followed this evolution - Be ready for the unforeseen - Data quality monitoring: larger processing power than anticipated together with a higher throughput data access - Use of IP-based clients of the cluster file system

21 21 ALICE Data Storage Architecture 180 ports FC4G 75 Transient Data Storage (TDS) storage arrays each divided in 3 volumes Nodes accessing data over FC4G: 75 data formatting and writing-only nodes (GDC) 30 reading-only nodes exporting data files to Permanent Data Storage (PDS) Nodes accessing data over IP 90 nodes for data quality monitoring Unique logical view by cluster file sharing! Event Building Network (Gbit Ethernet) GDC TDS GDC 75 GDC 30 TDSM TDSM Storage Network (FC4G) 75 TDS TDS TDSM DA DQM ACR 60 DA/DQM 30 ACR Archiving on Tape in the Computing Centre PDS

22 22 Data Recording and Formatting GDC 75 PCs based on 4-core Nehalem Linux RH SLC4 Characteristics CPU 2 E GHz Chipset FSB Memory PCI-X Format 12 GB DDR x (64 bit 133 MHz) 1 QLOGIC QLA FCS HBA 4 Gbps 1U Performance Max Event Building Bw Max Data Recording Bw 110 MB/s 390 MB/s

23 23 SAN Evolution over Time (hw deployment 10-40% performance): FC switches (16 FC4G, 4 FC10G) - Up to 30 nodes - Up 30 storage arrays FC4G 16 ports Storage Server FC4G 16 ports Storage Server FC4G 16 ports Storage Server FC4G 16 ports Storage Server

Server FC10G 4 ports FC4G 32 ports FC4G 16 ports FC4G 16 ports Storage Server Storage Server Storage

24 24 SAN Evolution over Time 2009: (hw deployment 100% performance) - 2 enterprises FC switches (8 slots, 16 FC4G, 4 FC10G) nodes over FC4G, 90 nodes over IP - 75 storage arrays Storage Server Storage Server FC10G 4 ports FC4G 32 ports FC4G 16 ports FC4G 16 ports Storage Server Storage Server Storage Server Storage IP relay FC10G 4 ports FC4G 32 ports Server FC4G 16 ports FC4G 16 ports Storage Server Storage Server

25 25 Storage Network Storage network: Fibre Channel switches FC 4G 2 enterprise FC4G switches 4 stackable FC4G switches

26 26 Transient Data Storage System Logical model: distributed file-system Hardware Transient Data Storage (TDS) Located at the experimental area Capacity: a day of autonomous data taking Before archiving to permanent data storage Permanent Data Storage Located in the computing centre (3 km away) Infinite capacity, very low cost TDS Software implementation TDS sharing between all nodes and applications

27 27 Storage Arrays Transient Data Storage (185 TB of RAID-6 disk buffer at the experimental area) Storage arrays 2 FC 4G ports 16 SATA II HD 250 GB

28 28 Use of a Cluster File System Requirements Maximize performance: absolute aggregate bandwidth performance (ALICE needs up to 3 GByte/s for write and read traffic) Minimize footprint required for the hardware equipment: the ALICE DAQ counting room is only 60 m 2 Scalability for the number of clients (100+ user nodes) Hardware neutral 2 Products tested in depth Both solid products. No product had any severe problem during these tests. Performance was the key selection criteria. Performance target: 1/5 of the global performance required: 300 MB/s writing combined with 300 MB/s reading The support given by the two companies during the installation and the tuning of the products was very good in both cases. Hardware virtualization Ensure high performance while adding flexibility

29 29 Cluster File System Tests (2) SN + TG SN

30 30 Affinities Cluster File System: Sharing of hundreds of disks by tens of PCs Global file system (system management) Minimal performance penalty Each stripe group has an affinity (assignment of data to specific storage class) Make file system across all Racks R global file space Add AFFINITIES Certain directories are bound to certain Containers C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18 C19 C20 C21 C22 C23 C24 R1 R2 R3 /filespace/global_file.raw /filespace/r3/c22/ /filespace/r3/c22/local_file.raw

Limited time to celebrate after the first collision A dramatic race

31 31 First pp Collision 23-Nov-09 First Physics Paper 01-Dec-09 Science is also moving faster and faster! Limited time to celebrate after the first collision A dramatic race between the 4 experiments for the first paper ALICE paper published only 8 days after data taking!

32 32 Being Prepared for the Heavy-Ion Run in November Data Writing Movers Data Archiving

33 33 LHC Upgrade LHC accelerator will be upgraded in 2020 and even more data will be produced A better selection mechanism and more accurate quality monitoring have to be put in place Data storage will also require higher performances and a faster access New technological progress will be required for the next steps of Physics!

34 34 Conclusion The performances promised by future technologies have been essential in the 90s to design the LHC accelerator and the experiment 4 GB/s of sustained data throughput 15 PB/year Successful integration of computing and storage equipment from several vendors Cluster file system provides the overall architectural coherency A new quantum leap is now needed for the LHC upgrade in a decade

A High-Performance Storage System for the LHCb Experiment Juan Manuel Caicedo Carvajal, Jean-Christophe Garnier, Niko Neufeld, and Rainer Schwemmer

658 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 57, NO. 2, APRIL 2010 A High-Performance Storage System for the LHCb Experiment Juan Manuel Caicedo Carvajal, Jean-Christophe Garnier, Niko Neufeld, and Rainer