The EKP Computing Model

Transcription

1 IEKP-KA/ The EKP Computing Model Volker Büge 1 (volker.buege@cern.ch) Armin Scheurer 1 (armin.scheurer@cern.ch) 1 Institut für Experimentelle Kernphysik, KIT April Introduction During the past years, the requirements concerning the computing infrastructure of the Institut für Experimentelle Kernphysik (EKP) have continuously evolved. This reflects the increased storage and CPU demands of recent HEP experiments, the grown number of people in the different EKP working groups and the amount of used computing infrastructures and facilities for end-user analyses. Having started with an installation in the 9 th floor of the physics building of KIT Campus South, the EKP computing infrastructure has been extended to a dedicated cooled room in the basement of the Gerthsen auditorium. With the increasing demand on computing power, additional resources at the shared Institute Cluster (IC1) at SCC Campus South have been acquired. Furthermore, the EKP is strongly involved in the experiment-specific administration of the WLCG Tier-1 centre GridKa at SCC Campus North where also storage and worker node installations for the German D-Grid community are available. Since the current computing concept has been designed for a single local installation, such a spread infrastructure requires a fundamental re-design. In the following, the current setup of the EKP computing infrastructure is briefly described, including all available resources at the different locations. Additionally, the requirements and workflows of the different EKP working groups are depicted which leads to a proposal for a combined, fully scalable and trend-setting computing model for the EKP. 2. Current Status The current available computing infrastructure comprises the following locations: EKP 8 th /9 th Floor: Central point is the air-conditioned server room 9-12, hosting mainly services re- 1

2 quired for the local desktop infrastructure, including about 60 desktop PCs, various notebooks, printers, print-server, web-server and monitoring infrastructure. A dedicated control machine provides a central user-space. Debian and Ubuntu have been chosen as operating system for the desktop cluster. EKP Basement: An air-conditioned server room in the basement of the Gerthsen auditorium, shared by different institutes, hosts our local development, worker node and storage infrastructure. Currently about 10 portal machines are available for development and debugging purposes for the users, accompanied by 45 cores for sequential and interactive end-user analysis. Due to the requirement of a certified analysis environment for the experiment specific software, Scientific Linux CERN/FNAL (SLC/SLF) or CentOS is installed on all portal machines. The actual version of the operating system is defined by the central computing and software groups of the different experiments. For the worker nodes, currently SLC 4 and 5 are in use. Moreover, a storage capacity of 80 TB spread over several file-servers is offered for user data to allow fast turn-around cycles. Due to historic reasons, an additional dedicated control machine provides a separated central user-space for the basement installations. All traffic from the 8 th /9 th floor and the basement is routed via the single EKP firewall towards the KIT network backbone. The whole infrastructure is administrated by Diploma and PHD students, leading to fast response times in case of failures, especially on weekends. SCC South IC1: The IC1 cluster with its 1600 cores and 380 TB of Lustre storage space is shared among several KIT institutions. The share for the EKP is about 350 cores and 150 TB of disk-space for local end-user analyses. It was planned to offer free capacities of this share to other German CMS collaborators via Grid means. The operating system chosen by the SCC administration was SuSE Linux Enterprise 10 (SLES10). As both, the analysis and the Grid environment require Scientific Linux CERN, the concept of operating system virtualization was a fundamental issue of the call. Unfortunately, due to severe problems with the implementations of the virtualization layer, a stable link to the WLCG Grid infrastructure could not be reached up to now. Also a dynamic partitioning with virtual worker nodes could only be realized in a test setup. A temporary workaround for the analysis environment under SLES10 allows a limited usage of the IC1 resources at all. An additional drawback, however, is that the specific software for the Belle experiment does not run under this operating system and thus prevents the usage of IC1 by the whole EKP Belle working group. Although the fast and scalable storage architecture of Lustre was designed to provide a large bandwidth for end-user analyses, various downtimes and Lustre instabilities have occurred during the past years, which embarrasses the IC1 to become a reliable and stable resource for EKP members. 2

3 SCC North GridKa: GridKa, as a WLCG Tier-1 centre, is responsible for the hosting of primary LHC storage and computing resources. In addition, several other HEP and astroparticle physics experiments are supported. They have long-term experience with the operation of large-scale sequential SLC Grid clusters in combination with the allocation of typical HEP storage infrastructures. Moreover, substantial Grid resources from the D-Grid community are available for German users. 3. Requirements Although the different user groups of the institute are involved in different experiments and analysis groups, their workflow could be virtually broken down to the following compromise, assuming about 60 active users: Storage Capacity: For the desktop and the cluster environment, the concept of three different storage classes is well accepted. Each user should have about: 2 GB of user space with daily backup, mainly used for login, settings and personal folders. 30 GB of home space with backup every three days, mainly used for code development and final results. 2 TB of scratch space without backup, mainly used for reproducible data skims. Moreover, a high available partition for hosting the virtual machine images as well as experiment-specific software is required. Computing Nodes: For fast turnaround cycles of serial event processing, about 80 computing cores accessible by a local batch system are needed. The batch system has to be capable of handling a large number of serial, non-standardized user jobs with an envisaged throughput for each core of about 5 to 10 MB/sec. As operating system, SLC is mandatory. The batch cluster has access to all storage areas. Portal Machines: For the actual code development and debugging, 10 portal machines running SLC/SLF/CentOS are required. The machines are intend to run as virtual instances with adaptions to the peculiarities of the different user groups. They have access to all storage areas. Grid Infrastructure: To provide the possibility for user Grid jobs to transfer results to local storage resources for further analysis, a Grid Storage Element (glite SE) is needed. A very high availability and long-term announcements of downtimes is mandatory and demanded by WLCG. Moreover, an interactive access to official datasets hosted 3

4 on such an SE would be advantageous to be fully included in the new EKP storage concept. 4. Computing Concept 4.1. Overall Having in mind the historically grown architecture, a change of the current used EKP computing model is indispensable. However, most of the changes can be realized by minor adaptations of the existing infrastructure. In the meantime, virtualization techniques have reached a level of performance which imposes their usage for all service machines including worker nodes. Such an abstraction layer from the underlying hardware offers a flexible, easy to administrate and high available solution to perfectly match the above discussed requirements. This will be completed by a change from individual file-server space to a global cluster filesystem, connected via a 10 GBit networking infrastructure. We aim for a unified storage solution offering the following storage classes, most transparent to the user: User and home space: all data stored will be replicated on different file-servers within the cluster filesystem in the 8 th /9 th floor and the basement. It will be available on the desktop cluster, the portals and the worker nodes. Scratch space: only one copy of the data is distributed over the storage resources in the basement. It will be available on portals and worker nodes. It is currently not planned to make this storage space available on the desktop cluster due to the limited network bandwidth between offices in the 8 th /9 th floor. VM space: all virtual images are replicated in the 8 th /9 th floor and the basement. Only VM host machines have access to this partition of the cluster filesystem. Local desktop space: the scratch directories of the local desktop machines are meant for short-term storage of end-user analysis files which could be interactively accessed on the desktop. Users should be aware that this storage space is not covered by a backup procedure. The details for each of the locations included in the new computing model, as sketched in figure 1 and 2, are defined in the following subsections EKP 8 th /9 th Floor Two standard server machines are sufficient to host all virtual service machines required by the desktop and notebook infrastructure, such as authentication (kerberos), printing (cups), Windows domain controller (samba/ad) and user management (ldap). Due to the importance of such services for daily work, a highly available virtual image management is indispensable and will be realized by a shared cluster filesystem between 8 th /9 th floor and basement. Despite the availability of an image clone at two different locations 4

5 ssh, nfs EKP Computing Concept Global View KIT Network Network Backbone Network Uplink Network Access World KIT IC1 WN RZ gsissh, nfs/4 srm, dcap srm, dcap GridKa Storage-Manager IC1 Lustre Storage Login-Server IC1 Tier3 Storage / WN D-Grid Storage / WN Tier1 Storage / WN Services Portals ssh, nfs/4 Firewall EKP Basement EKP VM Host Services EKP 8 th / 9 th Floor BladeCenter Workernodes Cluster FS Cluster FS Desktop Cluster Figure 1: Schematic overview of the EKP computing model, describing the locations EKP 8 th /9 th floor, EKP basement, RZ/IC1 and GridKa. for disaster recovery, this allows live migration of the services in case of maintenance periods as well. In a similar way, the availability and comfort of user and home directories will be increased. This however requires sufficient storage and network capacities at this location EKP Basement Due to scalability, power consumption, cooling efficiency and administration, a blade center infrastructure will replace the existing server and worker node based installation. All services will run as virtual machines. Having a blade center infrastructure hosting virtual machines allows the most flexibility in case additional computing resources are required. Moreover, the above described storage architecture would increase the reliability of the service and portal machines. In the field of batch system worker nodes, the usage of virtual machines provides an elegant mechanism to evaluate new operating system versions outside the production environment and guarantees a smooth transition as soon as the next version of e.g. 5

6 EKP Computing Concept Detailed View KIT Network Network Backbone Network Uplink Network Access World KIT Portals CMS (1...N) CDF (1...N) Other (1...N) LDAP/Kerberos What have I missed? Basement Mirror Firewall LDAP/Kerberos Samba/AD VM Host Windows Server EKP 8 th / 9 th Floor WebServices (www, Ganglia, Happyface, Trac,...) Workernodes BladeCenter Other Services Existing Infr. (Partial) Other Services VM Host Notebook Net /cluster/users /cluster/home /cluster/storage /cluster/vm Storage Classes [...] FS 1 FS 2 FS 3 FS N FS N+1 FS N+2 Cluster FS Desktop Cluster Figure 2: More detailed schematic overview of the EKP computing model, describing the locations EKP 8 th /9 th floor and EKP basement. SLC becomes mandatory. As the scratch storage class is only accessible from the portal machines and the worker nodes, the largest part of the file-servers for the storage architecture is based here. Due to the possibility of a frequent data exchange between the two locations SCC North and South, the basement has to be attached to the KIT network backbone with a high network bandwidth SCC South IC1 The already existing EKP share of the IC1 computing and storage resources will be used by dedicated analyses requiring high throughput and interactive login possibilities. For this, the current implementation of the batch system virtualization will be enabled for all IC1 worker nodes. Moreover, at least one portal machine acts as entry point for the Lustre storage space. Based on the experiences with the IC1 deployment, stability and availability during the last years, we do not plan to extend the resources at SCC South. 6

7 4.5. SCC North GridKa The existing resources of the D-Grid community for German users combined with the presences of important datasets at GridKa Tier1 complements perfectly the local infrastructure. As the currently available Grid-based access is accompanied by an enormous latency, an interactive login machine (storage-manager) is required to access the data in a comfortable and convenient way. This machine reflects a central component in the new EKP storage management and thus requires access to all available storage resources. The login authorization to this machine will be realized by existing Grid means and requires no further user management. A. Implementation Details A.1. Overview The following components have to be purchased or modified to fulfill the proposal made in this document, chronologically ordered. The list is visualized in figure Machine for the storage-manager located at SCC North GBit network backbone for the EKP basement. 3. Blade center chassis with a minimum configuration of blades for the startup, depending on available funds. 4. Upgrade of existing file-servers to 10 GBit network interfaces GBit network link between EKP 8 th /9 th floor and EKP basement. 6. Additional file-servers for the EKP basement. 7. Additional file-servers for the EKP 8 th /9 th floor. 8. Migration of current VM hosts from the EKP basement to the 8 th /9 th floor. A.2. Costing of Required Components (April 2010) Storage manager: Estimated Price: 5000 e Proposed Manufacturer: SysGen One potential 1U or 2U 19 server rack machine with two Quad-Core CPUs, 16 GB of memory, 2 TB of local disk space and 10 GBit ethernet connection. Network backbone: Estimated Price: e Proposed Manufacturer: HP A stackable, not managed switch architecture with a 1U 19 server rack design 7

8 Figure 3: Visualization of the acquirements required for the locations GridKa, EKP 8 th /9 th floor and EKP basement. provides the most flexibility for further extension when needed. models with 20 1 GBit ports and up to 4 10 GBit ports per rack. Required are Blade infrastructure: Estimated Price: e Proposed Manufacturer: HP HP BladeSystem c7000 enclosure with up to 16 blade device bays and 2 10 GBit uplinks to the network backbone. As a starting point, at least 2 blade servers, e.g. HP ProLiant BL465c Server series blades, are required to setup the infrastructure. File-server and upgrades: Estimated Price: e For the redundant and unified storage architecture, the most potent existing fileservers have to be upgraded with 10 GBit ethernet interfaces. In addition, 1 to 2 new file-servers are required to host the unified filesystem in the 8 th /9 th floor. 8

9 B. Acknowledgments Our thanks go to the whole EKP admin team, especially to M. Kreps and T. Kuhr for their wealthy input and discussion. Our special thanks go to F. Stober for setting up a prototype installation of GlusterFS on existing EKP resources and as well to the team of volunteers which helped in evaluating this storage solution for our computing concept. Many thanks to A. Heiss and A. Poschlad for the fruitful discussions regarding our planned installations as SCC Campus North. Additional thanks go to J. Ott and F. Ratnikov for providing information about typical usage scenarios as representatives for their working groups at the EKP. 9