ALICE GRID & Kolkata Tier-2

Transcription

1 ALICE GRID & Kolkata Tier-2 Site Name :- IN-DAE-VECC-01 & IN-DAE-VECC-02 VO :- ALICE City:- KOLKATA Country :- INDIA Vikas Singhal VECC, Kolkata

2 Events at LHC Luminosity : cm -2 s MHz every 25 ns 20 events overlaying

3 The Grid Computing Model Lab a Uni x ATLAS USA Lab m CERN Tier 1 UK Uni a CMS Tier3 physics department Tier2 Tier 1 Italy CERN CERN Tier 0 LHCb France Japan Uni n Desktop Lab b Scandinavia Germany Lab c Uni y Uni b Tier 0 Centre at CERN

4 ALICE computing model Online System RAW data delivered by DAQ undergo Calibration and Reconstruction which produce for each event 3 kinds of objects: 1. ESD object 2. AOD object 3. Tag object Online Farm This is done in Tier-0 site. Further reconstruction and calibration of RAW data will be done at Tier 1 and Tier 2. Tier 1 France Regional Center 10Gb/s Germany Regional Center Tier 0 Italy Regional Center ~40 Gb/s CERN Computer Center APROC Taiwan The generation, reconstruction, storage and distribution of Monte-Carlo simulated data will be the main task of Tier 1 and Tier /622 Mb/s Tier 3 Tier 2 Kolkata 1-10 Gb/s Tier2 Center Tier2 Center Tier2 Center Tier2 Center Physics data cache Institute Institute Institute Institute Mb/s DPD (Derived Physics Data) objects will be Processed in Tier 3 and Tier 4. Tier 4

5 Size: 16 x 26 meters Weight: 10,000 tons HMPID ALICE Setup LHC Utilization -- ALICE TOF TRD TPC PMD ITS PHOS Muon Arm Indian contribution to ALICE : PMD, Muon Arm

6 ALICE Collaboration ~ 1/2 ATLAS, CMS, ~ 2x LHCb ~1100 people 30 countries, 80 Institutes Total weight 10,000t Overall diameter 16.00m Overall length 25m Magnetic Field 0.4Tesla The ALICE collaboration & detector

7 Data volumes RAW data 2.5 PB/year Two distinct periods p+p (~7.5 months) and Pb+Pb (~40 days) Reconstructed and simulated data 1.5PB first level RAW filtering (ESDs) 200TB second level RAW filtering (AODs) 1PB of simulated data User generated data ~500TB Total ~5 PB of data per year (without replicas) Replication 2x RAW, 3x ESD/AODs, 2x user files Taken from L. Betev Slides in T1-T2 Meeting at Karlsruhe during Jan 2012

8 Processing RAW data reconstruction ~10K CPU cores MC processing ~15K CPU cores User analysis ~7K CPU cores (450 distinct users) ~40Mio jobs per year ~ 1.3 job completed every second ½ production, ½ user jobs 200 Mio files per year Taken from L. Betev Slides in T1-T2 Meeting at KIT Karlsruhe during Jan 2012

9 KOLKATA ALICE

10 ALICE Sites on MONALISA Europe Africa South America 72 active computing sites Asia North America

11 Why Tier 2? 1. Tier-2 is the lowest level to be accessible by the entire collaboration. 2. Each sub-detector of ALICE has to be associated with minimum Tier-2 because of large volume of calibration and simulated data. 3. PMD is one of the important sub-detectors of ALICE. 4. We are solely responsible for PMD from conception to commissioning.

12 Grid Site As per WLCG & Experiment Requirement WMS MyProxy VOMS. Site BDII CREAM-CE WNs (More and More WNs) SE (PureXrootD) LCG-UI Disks (More and More Disks)

13 KOLKATA or General Site Central Services WMS MyProxy VO-BOX Cooling, UPS Fire Alarm, Access Control etc Monitoring Server Site BDII CREAM-CE DPM PureXrootD XrootD Redirector XrootD Disk Server Installation,DHCP Server etc.. Local and Global Network / Fiber Line from Network NFS SERVER PBS SERVER DNS SERVER UI SERVER Tier3 Manage ment Server and cluster HA SERVER Blade 64 bit Servers With Blade Enclosures 32or64bit Servers Few Tower Servers 1U & 2U Servers New SAN Box Old NAS Older NAS HP DELL IBM Etc Even Older DAS Disks Arrays (More and More Arrays)

14 Frontend component of Site & Installation VO-BOX Site BDII LCG-CE CREAM-CE Grid middleware meta-packages installed through YUM and configured through YAIM. Middleware changed time to time like GLITE EMI. (follow manual) During Kolkata Site installation and configuration we experienced about RPM dependencies with JAVA, Security packages etc. SE PURE XrootD LCG-UI Community and mailing list helps a lot. For most of the problem we got the solution from mailing list. Thanks to APROC, Taiwan for helping at each stage

15 Middleware installed on IN-DAE-VECC-02 Site 1.Installed SLC 5.8 (x86_64) operating system on x86_64 Machine. 2. Upgrading below middleware packages to EMI middleware. glite-vobox CREAM-CE (64bit) glite-bdii Pure XROOTD Redirector as Storage Element glite-wn (64bit) grid01.tier2-kol.res.in gridce02.tier2-kol.res.in dcache-server.tier2-kol.res.in For 79 Worker Nodes (476 core) wn045-wn123.internal.tier2-kol.res.in

16 Backend Component of SITE Router & Switch 2 networks, one Public Network and another Private network. Domain Name Server DNS server is critical component. We have 2 redundant Name servers Naamak & suchak for High Availability. Time Server Configured NTP protocol Installer Using Network installation and Automated configuration Quattor like tools. Storage Server Using NFS mounted Common shared space PBS Server CE & PBS batch scheduler on a Server. Configured Firewall (through iptables) and did NAT ing on it. TIER-3 Cluster Separate cluster for local users with Interactive and non interactive nodes. Monitoring Server Configured MRTG (Network Traffic Monitoring) and cluster monitoring tool.

17 Doing Preventive Maintenance Once in a Year

18 x.x (Stand by) x.x (Stand by) KOLKATA Grid Kolkata TIER-2 centre logical diagram 300Mbps Internet Router 1Gbps Fiber Backbone xx/27 Switch grid Grid-peer gridse001 naamak suchak Installer gridce02 grid01 SINP Backup-server wn001 wn TB Backup Switch-1 wn002 Switch-2 dache-server Switch-1 wn046 Switch-2 Computing Nodes wn Nodes Dell and Wipro Blades Cluster with 25 TB of As Tier-3 4 Xrootd Disk Servers Consisting of 230 TB of IBM And HP SAN system Computing Nodes wn122 DELL and HP Blade Server with Multi Core Xeon 3.0 GHz wn025 GRID-PEER Tier-3 cluster with 32 & 64 bit machine wn123 IN-DAE-VECC-02 Site with 64 bit machine

19 ALICE Tier-2 Grid Started in 2002 CERN 512Kbps Ethernet Bandwidth S. K. Pal & T. Samanta Started in Operating System Scientific Linux 3.05 Middleware Alice Environment with PBS as batch system Hardware (CPU, Disk) 1xDuel Xeon,4GB Compute Node 2xDuel Xeon,2GB WNs 2x80GB Disk Space Bandwidth 512Kbps Shared

20 From 2 Core to 700 Cores Started with Desktop Machine Tower Like Servers HP 1U Servers Wipro 1U Servers Single Core HP Blades Dual Core HP Blades Quad Core Dell Bladed Dual Processor Dual Core GPU Server with Tesla 2070 with 448Cores 2012

21 Kolkata Tier2 on Monalisa

22 From 512MB Disk to 300TB Disk Started with MB in Desktop Machine GB in Tower Like Servers as DAS GB in HP MSA TB Wipro NAS TB HP EVA SAN TB i-scsi TB IBM DS TB Hard disk in GPU Server 2012

23

24 From 128Kbps to 1Gbps Disk Started with Kbps shared link Kbps Mbps Dedicated Link Mbps from Bharti Mbps from Reliance Mbps from VSNL (ERNET) Mbps from NKN Upgrading with 1Gpbs 2012

25 Efficient Cooling Concept and Implementation Hot and Cool Air is separated. For air separation, Cold Air Containment is created. Cold Air Containment is least accessible Area. Cool only hardware racks, not human, walls etc. Human intervention to Cold Aisle Containment is restricted. All the management and monitoring of the server, storage is from outside Cold Aisle Containment. All the power and Ethernet cables are also from outside Cold Aisle Containment. Temperature gradient between Cold and Hot aisle is 5 o C

26 Kolkata Tier-2 After renovation

27 Major Achievements Consistently more than 400 ALICE Jobs are running after Commissioning of the efficient Cooling Solution.

28 Achieved pledged resources Kolkata Tier-2 provided total 6.0K HEP SPEC2006 CPU and 230TB of Disk Storage.

29 Jobs completed KOLKATA Grid 1M ALICE Job completed during Last Year Performance: ~1M jobs successfully completed during last one year Time ->

30 Total Kolkata Tier-2 Resources Computing Resources:- Total :- 476 Cores DELL Blades 32 * 8 = 256 HP Quad Core Blades 8*8= 64 HP Dual Core Blades 39 * 4 = 156 Storage :- 230TB under one HP 2U Management Server 74TB : HP EVA 6100 under 2 * 2U HP disk server 156TB : IBM DS 5100 under 2 * 1U IBM disk server 300Mbps Network speed. It will be increased upto 1Gbps during this year.

31 After NKN Network, Speed Increased to 300 Mbps

32 Grid-Peer Tier-3 Cluster 1U Sliding LCD Monitor with 16 port KVM Dell(TM) PowerEdge(TM) M1000e Blade Server Chassis. 16 Number of Dell(TM) PowerEdge(TM) M610 High Performance Intel Blade Each blade has latest Nehalem based 2 * Intel Quad Core E5530 Xeon 2.4GHz CPU with 8MB cache. Each blade has16gb RAM. Each blade has 2 * 146GB Mounted as RAID1. Installed SLC 5.6 x86_64 OS (kernel version el5). Dell ISCSI EqualLogic Storage 16 * 2TB SAS hard disks TB Usable space after RAID5 and Hot Spare.

33 Grid-Peer Tier-3 Cluster cont Total 25 Nodes for VECC users and PMD Collaborators bit nodes 13 64bit computing nodes 32 bit nodes are on oldest hardware procured in 2004 (slowly we will deprecate them as High noise, power and Heat Generation.). 25 TB of Total storage active users (across India.) 30 + active users (in VECC.) Quota implemented. Root, Geant3, Aliroot, Alien, Fortran etc user specific software installed according to hardware like 32 bit and 64 bit. Extensively used by the users, need to extend.

34 Bi-product of WLCG GRID Intra-DAE Grid EU-India Grid Health Grid IGCA GARUDA Grid

35 Thank You

36 Supporting Slides

37 Main data types in ALICE Raw data Event Summary Data Pass1 T0 Conditions Calibration Alignment data OCDB (updated by pass0 -passn AliEn FC ESD AliRoot RECONSTRUCTION filtering Event Summary Data Event Summary Data AOD standard Pass2 T1 PassN T1 Analysis Analysis Analysis Monte Carlo + extra ESD run/event numbers, trigger word, primary vertex, arrays of tracks/vertices, detector info AOD standard cleaned-up ESD s, reducing the size by a factor of 5 Can be extended on user demand with extra information ESD and AOD inheriting from the same base class (keep same event interface)

38 ALICE Job Catalogue Job 1.1 Job 21.2 Job 31.3 Job 2.1 Job 2.1 Job 3.1 Job 3.2 lfn1, lfn2, lfn3, lfn4 lfn1, lfn2 lfn2, lfn3, lfn4 lfn1, lfn3, lfn2, lfn4 lfn3 lfn1, lfn3 lfn2, lfn4 lfn1, lfn3 lfn2 Optimizer Registers output Close SE s & Software Matchmaking KOLKATA Tier-2@Alice Grid Job submission Asks work-load Submits job ALICE File Catalogue lfn guid {se s} lfn guid {se s} lfn guid {se s} lfn guid {se s} lfn guid {se s} User Job ALICE central ALICE catalogues services Site Execs agent Yes User Env OK? VO-Box LCG No Die with grac e Updates TQ Retrieves workload Receives work-load Sends job result AliEn CE packman WMS Sends job agent to site CE WN

39 Xrootd architecture Global redirector (not in picture) intra-site storage collaboration Redirectors Cache file location open file X 2 nd open X Redirector (Head Node) go to C Who has file X? go to C A B Client Client sees all servers as xrootd data servers All storages are on WAN C Data Servers Cluster

40 Grid security (in a nutshell!) Important to be able to identify and authorise users Possibly to enable/disable certain actions Using X509 certificates The Grid passport, delivered by a certification authority. (IGCA for India) For using the Grid, create short-lived proxies Same information as the certificate but only valid for the time of the action Possibility to add group and role to a proxy Using the VOMS extensions Allows a same person to wear different hats (e.g. normal user or production manager) Your certificate is your passport, you should sign whenever you use it, don t give it away! Less danger if a proxy is stolen (short lived)

41 The VOBOX The VOBOX is a WLCG service developed in 2006 to provide the experiments with a service to: a) Run their own services. b) In addition it also provides file system access to the experiment software area. The concept of VOBOX is not the same for the 4 LHC experiments a) ALICE requires the STANDARD WLCG VOBOX

42 SRM SRM SRM SRM KOLKATA Grid Storage strategy SE head node xrootd (manager) xrootd (worker) Disk xrootd (worker) DPM Old implement ation WN xrootd (worker) Castor Current version MSS DPM, CASTOR, dcache are LCGdeveloped SEs, xrootd is entering as a strategic solution xrootd emulation (worker) MSS dcache Working, but severe limits with multiple clients

43 What is MonALISA? Caltech project started in Java-based set of distributed, self-describing services Offers the infrastructure to collect any type of information Can process it in near real time The services can cooperate in performing the monitoring tasks Can act as a platform for running distributed user agents

44 MonALISA software components and the connections between them Clients HL services Data consumers Proxies Multiplexing layer Helps firewalled endpoints connect Agents MonALISA services Data gathering services Network of JINI-Lookup Services Secure & Public ully Distributed System with no Single Point of Failure Registration and discovery

45 PROOF KOLKATA Grid Parallel ROOT Facility Interactive parallel analysis on a local cluster Parallel processing of (local) data Fast Feedback Output handling with direct visualization PROOF is part of ROOT

46 PROOF Schema Client Local PC Result Remote PROOF Cluster root stdout/result ana.c root node1 Result ana.c Data root Data node2 root node3 Result Data Proof master Proof slave root Result node4 Vikas Singhal, VECC, Data INDIA