Summer Student Project Report

Transcription

1 Summer Student Project Report Dimitris Kalimeris National and Kapodistrian University of Athens June September 2014 Abstract This report will outline two projects that were done as part of a three months long summer internship at CERN. In the first project we dealt with Worldwide LHC Computing Grid (WLCG) and its information system. The information system currently conforms to a schema called GLUE and it is evolving towards a new version: GLUE2. The aim of the project was to develop and adapt the current information system of the WLCG, used by the Large Scale Storage Systems at CERN (CASTOR and EOS), to the new GLUE2 schema. During the second project we investigated different RAID configurations so that we can get performance boost from CERN s disk systems in the future. RAID 1 that is currently in use is not an option anymore because of limited performance and high cost. We tried to discover RAID configurations that will improve the performance and simultaneously decrease the cost. 1 Information-provider scripts for GLUE2 1.1 Introduction The Worldwide LHC Computing Grid (WLCG, see also 1) is an international collaboration consisting of a grid-based computer network infrastructure incorporating over 170 computing centres in 36 countries. It was originally designed by CERN to handle the large data volume produced by the Large Hadron Collider (LHC) experiments. This data is stored at CERN Storage Systems which are responsible for keeping and making available more than 100 Petabytes (105 Terabytes) of data to the physics community. The data is also replicated from CERN to the main computing centres within the WLCG. With such a diversity of sites and different storage systems having a common information system is crucial to guarantee interoperation among the different computing centres. The current WLCG information system conforms to a schema named GLUE which evolving towards a new version: GLUE2. The specific work we had to do was to develop and adapt the current information system used by the Large Scale Storage Systems at CERN (CASTOR and EOS) to the new GLUE2 schema. Reader can check references 2, 3 and 4 for more information about GLUE/GLUE2. 1

2 1.2 Information Providers for CASTOR and EOS The information system of CERN s Storage Systems is expected to provide the user with data i.e. total storage space used by some Virtual Organization (VO), how much of this storage space is online and how much is nearline, the protocols the user can use to access the data, etc. The user can ask for this information through LDAP (standard application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network) search. There are two monitoring boxes for CASTOR and two for EOS. The information providers are perl scripts running inside those boxes and they are responsible for fetching all the required information from web pages and Service Level Status (SLS). As long as they gather the required data they print an LDIF (a standard plain text data interchange format for representing LDAP) file in a way compatible with GLUE schema. What we had to do was update the perl scripts so that they would display the information in a way that will be compatible with GLUE2 schema. Information providers work in two levels: 1. Gather information for CASTOR and EOS services in the way mentioned before. 2. Arrange this information in classes and print one LDIF file with the attributes of every class. There was no need to change the way that we collect the information. What we did was organizing it under a different structure. By that we mean that some classes that existed in GLUE do not exist anymore in GLUE2 (e.g. GlueSEControlProtocol), some were splitted in several others (e.g. GlueSE is splitted in GLUE2StorageService and GLUE2StorageServiceCapacity), GLUE2 has new classes as well (e.g. GLUE2StorageManager), etc. We firstly designed and implemented the architecture of the new classes. After that we pass the information that we collect in this structure and we print the LDIF file which is compatible with GLUE2 schema. Before we put the new information providers into production we tested that the LDIF file was correct under GLUE2, using a tool called GLUE validator, which confirmed the correctness of the output of the scripts. Next step was the integration of the information providers in production. We deployed the new scripts (GLUE2) in the monitoring boxes and run them in parallel with the old ones (GLUE), because we want be able to publish both GLUE and GLUE2. After some finals tests about the correctness of the LDIF files in production CERN s Storage Systems can now publish their information also in GLUE2. The copmlete transition from GLUE to GLUE2 is expected to happen in the future. 1.3 Reuse of this work As a conclusion we would like to add that there are more sites outside CERN running CASTOR/EOS (e.g. RAL, Taiwan). Despite the fact that the scripts that we developed for this project are CERN specific, with a little effort we can 2

3 adapt them to the singularities of every site and the transition from GLUE to GLUE2 for these sites can be done easily. 2 RAID Configurations 2.1 Introduction RAID (Redundant Array of Inexpensive Disks, see also ref 1) is a concept that was developed in 1977 by David Patterson, Garth Gibson, and Randy Katz as a way to use several inexpensive disks to create a single disk from the perspective of the OS while also achieving enhanced reliability or performance or both. As the disks of the array are independent you can read/write data from/to several disks at a time and this improves the performance. Apart from that in almost all RAID levels (except from RAID 0) there is a way to recover the data after a disk failure and avoid an array failure. This offers data reliability. There are 7 different standard levels of RAID: 1. RAID 0: splits data across drives, resulting in higher data throughput. The performance of this configuration is extremely high, but a loss of any drive in the array will result in data loss. This level is commonly referred to as striping. 2. RAID 1: writes all data to two or more drives for 100% redundancy: if either drive fails, no data is lost. Compared to a single drive, RAID 1 tends to be faster on reads, slower on writes. This is a good entry-level redundant configuration. However, since an entire drive is a duplicate, the cost per megabyte is high. This is commonly referred to as mirroring. 3. RAID 2: stripes data at the bit level instead of the block level (remember that RAID-0 stripes at the block level) and uses a Hamming Coding for parity computations. In RAID-2, the first bit is written on the first drive, the second bit is written on the second drive, and so on. Then a Hammingcode parity is computed and either stored on the disks or on a separate disk. 4. RAID 3: uses data striping at the byte level and also adds parity computations and stores them on a dedicated parity disk. 5. RAID 4: uses data striping at the block level and also adds parity computations and stores them on a dedicated parity disk. In a similar fashion, RAID-4 builds on RAID-0 by adding a parity disk to block-level striping. 6. RAID 5: stripes data at a block level across several drives, with parity equality distributed among the drives. The parity information allows recovery from the failure of any single drive. Write performance is rather quick, but because parity data must be skipped on each drive during reads, reads are slower. The low ratio of parity to data means low redundancy overhead. 3

4 (a) RAID 10 array. (b) RAID 60 array. Figure 1: Hybrid RAID arrays. 7. RAID 6: is an upgrade from RAID-5: data is striped at a block level across several drives with double parity distributed among the drives. As in RAID-5, parity information allows recovery from the failure of any single drive. The double parity gives RAID-6 additional redundancy at the cost of lower write performance (read performance is the same), and redundancy overhead remains low. Note that RAID 2,3,4 are not in use anymore. There are nested RAID levels too, also known as hybrid RAID, which combine two or more of the standard levels of RAID to gain even better performance, additional redundancy, or both. Examples: RAID 1+0 or 10, 5+0 or 50, 6+0 or Hardware and Software RAID arrays There are several operations that take place in a RAID array, such as sending chunks of data to the appropriate disks, computing parity, hot-swapping, disk fail-over, checking read transactions to determine if the read was successfully and if not, declaring that disk as down and more. All of these tasks require some sort of computation and have to be performed by a RAID controller. There are two options for RAID controllers: 1. hardware RAID that has a dedicated RAID controller to run the RAID application. Hardware RAID has very good performance but it is expensive and the RAID controller is a single point of failure. If it fails the whole RAID array fails. 2. software RAID that uses the CPU for RAID chores without need for any additional hardware. Software RAID is cheaper but the performance is worse and processing load is added on the CPU. During the last years the gap in performance between hardware and software RAID is closing and the flexibility and reliability offered by software RAID made it the first choice for many systems. 2.3 Creating and testing software RAID arrays During our performance tests we focused on two RAID levels that fitted our needs, i.e. high performance, good performance/cost ratio, high fault tolerance and good rebuild performance. Those were RAID 10 and RAID 60 (figure 1). The basic reasons that led into excluding other RAID levels were: 4

5 RAID 0: no fault tolerance at all. RAID 1: same cost as RAID 10 but much worse performance. RAID 5: if one of the HDD of the array fails then you have to read all the rest of the HDDs without any error to rebuild it. Otherwise the whole array will fail. RAID 50: it connects several RAID 5s and that makes it vulnerable in the way that we mentioned above for RAID 5. RAID 6: you need many HDDs to get a big performance boost and that limits data security. Let us symbolize RAID x/y(/z): x: RAID level. y: total number of disks in the array. z: number of standard RAID level arrays that we have if the array is nested. For example RAID 60/12/2 is a RAID 60 array with 12 disks in total and 2 RAID 6 arrays (of 6 disks each). We tested the following RAID 10 and RAID 60 configurations (inside the brackets is the redundancy of the configuration): RAID arrays: 10/12/6 (50%), 60/12/2 (33.3%), 60/16/2(25%), 60/24/3 (25%), 60/24/2 (25%). In all the arrays we used software RAID and were created with mdadm (see ref 2) and benchmarked with IOzone (see ref 3). The tests that we performed with IOzone measured: Write/Re-write performance. Read/Re-read performance. CPU utilization. of the RAID array for files of size: 1, 2, 4, 8, 16, 32, 64, 128GB. We took measurements when we had only 1 thread and when we had 10 threads running in the array. Furthermore, we measured the degradation in the array s performance while it was in rebuild mode. The machines that we performed the the tests in have the following characteristics: 64GB RAM 48 HDDs 4TB/HDD 5

6 Figure 2: Comparison for RAID 60s, 1 thread. 2.4 Results For small file sizes we notice very high performance due to buffer caching of the system (figures 2, 3). The real disk performance is much lower. After the caching effects are over we can notice that bigger arrays have greater speedup because writing/reading happens in many disks simultaneously. For 10 threads the ranking of the RAID arrays remains the same in terms of performance but the actual performance is degraded a lot because of many clients asking blocks from the array simultaneously. When we compare RAID 10 with RAID 60 we see that in reading there is no diference and the performance of the arrays is simply proportional to the number of disks they contain (see figure 4, for RAID 10 and 60 with 12 HDD each). In writing though there is a big gap in performance because RAID 10 does not need to compute any parity and just writes blocks in all disks simultaneously while writing in RAID 60 is more time consuming. 6

7 Figure 3: Comparison for RAID 60s, 10 threads. 7

8 Figure 4: Comparison for RAID 10 and 60, 1 thread. 8

9 2.5 Results for RAID 60 during rebuilding In RAID 60 a single HDD failure leads to reading the whole RAID 6 array where this disk belongs to. So, due to rapid increases in disk capacity and the slow improvement in disk access speed, RAID system are facing much longer disk reconstruction times than before, which can also reduce system reliability. Figure 5: Normal vs rebuilding for RAID 60, 12 disks, 2 RAID 6s. On the other hand the performance is not that much degraded during the rebuilding of the array, as we can see in figure 5. We notice 50% decreasing in performance in reading. During writing the performance is decreased by slightly less, around 20-30%. Despite the fact that the performance is still in accepted levels the process of rebuilding lasts for a long time. With very intensive requests to the server that happen when we run IOzone tests we had a rebuild time of days after only one single disk failure in RAID 60/24/3. In reality the load of the requests from clients in the server is never expected to be that high as with IOzone requests but still the rebuilding period can be several days for only one disk. 9

10 (a) Writing performance while rebuilding (b) Reading performance while rebuilding Figure 6: RAID configurations are ordered in the same way while rebuilding. 2.6 Ways to improve rebuild times Storage systems in CERN are about to use really big arrays so the reconstruction process is expected to take long. This is obviously a problem. If reconstruction takes too long then the probability of having a second disk failure increases because the load on the disks is much higher than usually due to the rebuilding process. There are two major ways to deal with it if we notice that the array is at risk. 1. Remove the machine from production. This will lead to 0 client requests and the array will be fully reconstructed in only some hours. 2. Linux kernel limits the speed imposed on the RAID reconstruction. In urgent cases you can speed up the process of reconstruction by increasing the minimum guaranteed speed of the rebuild of the array. This method 10

11 though, will place a much higher load on the system and will degrade the performance so it should be used with care. Code for decreasing the rebuilding time: Display maximum and minimum speed guaranteed for your system: > cat /proc/sys/dev/raid/speed limit max > cat /proc/sys/dev/raid/speed limit min 1000 Increase minimum speed guaranteed: > echo > /proc/sys/dev/raid/speed limit min I would like to add that from the results that we got we can see that we cannot continue increasing the size of the arrays otherwise we will face reconstruction times of weeks as disks will go larger. We have to find alternative solutions if we want extra performance for our systems. 2.7 Conclusion From the tests that we performed we got results that are compatible with theory. The most obvious is that the bigger the speedup we want, the bigger the array that we have to use. As for the configurations we got that RAID 10 is the best in terms of performance and rebuild times. RAID 60/24/3 seems to be the best among the RAID 60 configurations, and it is a good compromise between cost and performance. There are of course more tests to be done like: Rebuild tests: Rebuild time and performance with more than one failed disks and in several of the RAID 6 arrays that consist the RAID 60. Rebuild time and performance when we have less intensive requests than with IOzone. Rebuild time, performance and CPU utillization when you increase the default speed imposed on the RAID reconstruction. Check how the machines behave as a whole with many RAID arrays, and not only how a specific array performs. Finally I would like to say that before we apply a RAID configuration into production we have to think if we need all the speed-up that it can offer. Depending on the server we can have many small arrays (if we cannot exploit huge boost in performance for reasons like net bandwidth, or simply we do not need that great speed), or fewer bigger arrays for great performance guarantees. In the near future it is possible that many clients of CERN s servers (tapes, etc.) will start performing in much higher speed than they do now so the extra boost from the RAID arrays that today seems meaningless would be a must. 11

12 3 References Information providers for GLUE and GLUE2: 1. WLCG. 2. Glue Schema Specification. 3. Glue2.0 Schema Specification. 4. GLUE2 Template for DPM Storage Service RAID arrays: 1. RAID levels. 2. Software RAID with mdadm. 3. IOzone documentation. 4. Increase the speed of rebuilding. 5. Large disks, the end of RAID? 12