CMS Tier-3 cluster at NISER. Dr. Tania Moulik

Transcription

1 CMS Tier-3 cluster at NISER Dr. Tania Moulik

2 What and why? Grid computing is a term referring to the combination of computer resources from multiple administrative domains to reach common goal. Grids tend to be more loosely coupled, heterogeneous, and geographically dispersed as compared to conventional cluster computing. Grids are often constructed with the aid of generalpurpose grid software libraries known as middleware (glite, globus, in our case).

3 Grid vs Supercomputer The primary advantage Each node can be purchased as commodity hardware, which, when combined, can produce a similar computing resource as multiprocessor supercomputer, but at a lower cost. The primary performance disadvantage various processors and local storage areas do not have high-speed connections. Grid is well-suited to applications in which multiple parallel computations can take place independently, without the need to communicate intermediate results between processors. It can be costly and difficult to write programs that can run in the environment of a supercomputer, which may have a custom operating system, or require the program to address concurrency issues. If a problem can be adequately parallelized, a thin layer of grid infrastructure can allow conventional, standalone programs, given a different part of the same problem, to run on multiple machines. This makes it possible to write and debug on a single conventional machine, and eliminates complications due to multiple instances of the same program running in the same shared memory and storage space at the same time.

4 Grid computing versus Supercomputer Tier-1 Tier2 Tier-2 Tier-3 Tier-3 Tier-3 Processor 1 Processor 2 Processor 3 Computer Bus

5 Why distributed computing? Huge amount of data Collisions every 25 ns (40 MHz) Event data recording (High Level Trigger farm input buffer) ( Hz == 5 ms) 1 event = 2-3 Mbytes. So, 2-3 Billion events 2-6 Petabytes of data per year. Many institutions distributed over many countries Need fast transportation of data. 3 Tiered system Tier-0 : Raw data is collected and stored and then shipped to the Tier-1 centers. Data transfer from HLT to Tier-0 must happen in real time ~225 Mb/s Minimal data processing Tier-1 : Six Tier-1 centers distributed in various countries where the data is shipped for "custodial storage (Live copy of data). Processing and re-processing of data as well. Tier-2 : Pulls up the processed data from Tier-1. Could do further reprocessing depending on physics goals. Tier-3 : Reduced Dataset for individual analyses. Compute nodes house CMSSW specific software Batch submisson etc.

6 The CMS data distribution design Tier-1 : ASCC (Taipei), CCIN2P3 (Lyon), FNAL (Chicago), GridKA(Karlsruhe), INFN-CNAF(Bologna), PIC(Barcelona), RAL(Oxford)

7 Hardware configuration 2 U Server Rack mountable. 4 hot pluggable compute nodes Processor : 2 x Intel Quad Core 2.4/2.5 GHz with 12 MB Cache Chipset : Intel 5000P series chipset with 1333 MHz FSB Memory : 32 Gbytes ECC registered DDR MHz DIMMs Hard Disk : 2 x 250 GB Serial ATA Disk Drives, 7200 RPM, Hot swappable Interconnect : 2 x 1 TB SATA2 10k Hot plug HDD Network : At least 4 gigabit Ethernet ports Storage Space : XX (to be decided) Tbytes HDD based Raid Array 4-6 User nodes (Desktops/Laptops To be decided) Network NKN (?), Airtel Network switches, UPS, Cooling units, etc.

8 A typical Tier-3 Cluster Shared file system NFS server. Condor batch queue - Nodes in a batch queue managed wth Condor Interactive nodes The user client machine is an interactive node that allows users to login and provides them with software to access Grid services. Compute Element Sharing your local resources with other grid users. Storage Element e.g. raid array providing high performance data transfers over the grid.

9 Software configuration Basic OS - Scientific Linux 5.4 Cluster installation Using a Cluster management file e.g. ROCKS Using a Kickstart file Job Scheduling Condor Cluster Monitoring and Job throwing Ganglia. GUMS -- An optional Grid Identity Mapping Service that provides automatic capabilities for managing user lists as well as enabling users to have additional "groups" and "role" privileges.

10 User, compute and storage elements Use pacman Package management system to install most of OSG software. source /nfs/osg/app/pacman/setup.sh Gridclient software - software to access Grid services Worker Element : source /nfs/osg/app/pacman/setup.sh mkdir /nfs/osg/wn ln -s /nfs/osg/wn /osg/wn cd /osg/wn pacman -get ln -s /etc/gridsecurity/certificates /osg/wn/globus/trusted_ca Storage element : Storage Resource Manager (SRM) A GridFTP server - GridFTP is the standard way for moving large datasets across the grid, and is required for data subscription models. OSG Bestmangateway to install SRM+GridFTP

11 Tentative timeline Acknowledgement : Many thanks to TIFR group experts (Mr. P.V. Deshpande) Budget preparation for Tier-3 cluster is ready. Submit to India-CMS for approval after internal review and approval of director. Acquire hardware by early 2011 Software installation by mid 2011