Infomart ebook Series. A Sustainable Approach to Eliminating the Cause & Effects of Human Error in Data Centers

Transcription

1 Infomart ebook Series A Sustainable Approach to Eliminating the Cause & Effects of Human Error in Data Centers

2 Our new Constitution is now established, everything seems to promise it will be durable; but in this world, nothing is certain except death and taxes...and the inevitability of downtime in data centers due to human error. BEN FRANKLIN, DATA CENTER VISIONARY

3 What is the leading cause of downtime? In a connected world were everything is always on, 100% uptime is expected. Many data center operators do quite well. However, downtime still happens. So, what is the leading cause of downtime? Is it natural disasters? Looking at the RFPs we receive, you'd think that volcanoes, floods, and earthquakes were common events. How about external threats? Espionage? Hacking? RFP questions about fences, mantraps, guards, and security features are extensive. These are serious issues and deserve attention. But, according to 451 Research, 75% of all data center downtime is caused by human error.

4 Mitigating human error is the key to data center reliability. Historically, most data center designers and equipment providers have focused on reducing downtime through increasing levels of redundancy (e.g. the Uptime Tier ratings) and through producing more reliable equipment. In many ways, better design reduces vulnerability to many types of faults, including human error. Or more precisely, the effects of human error are mitigated through higher redundancy. We will discuss the serviceability and maintainability aspects of design, but most of this presentation will address the other parts of the operating equation, namely, people, strategy and the integration of processes and tools. Design Objectives People Strategy Process & Tools

5 Design Objectives Designs are often about choices. Much of the emphasis is on cost. Lowest first cost vs. lowest unit cost. Trading material cost for labor cost. Other priorities can be about reducing project development time. Assuming cost and time are equal, and assuming an expected level of quality (which is very high in the data center sector) we use the following principles:

6 Design for simplicity and maintainability. Choose industry standard components. Designing a custom quantity of capacity has hidden costs and risks. Using standard size increases maintainability, simplifies commissioning, re-calibration, and eventually replacement aspects. Physical segregation of services. Ask any system administrator how difficult change management becomes when multiple parties have competing authorities to make changes. This source of error can be addressed by physical segregation of services. Separate the data rooms from electrical rooms and mechanical equipment areas. This is the easiest way to gain better visibility and control of change. Open monitoring frameworks. The lifecycle of your monitoring system has to mirror the lifecycle of the data center forever. Closed systems are harder to maintain, change, and add new components. Customers and vendors benefit from the transparency. Data centers may have custom components, but the process of maintenance is not unique to data centers. We view ourselves as expert users, not software developers. Lifecycle of replacement. Nothing goes into a data center without a maintenance plan, which has to include a replacement plan. If the vendor can't offer a 5 year warranty or long term service contract, you will be insourcing maintenance or performing replacement. Eliminating fragile artifacts. Without any sacrifice of safety or functionality, some sources of downtime can be removed almost entirely. The image on the right is that of a component level EPO button. The feature is included by default, and it's meant to provide an easy way for an untrained person to shut down the device. There is a main breaker on the same panel which has identical functionality. In a data center where electrical equipment is segregated (and maintained by trained staff) this button isn't necessary and probably should be removed by the manufacturer or distributor. Beware, there may be jurisdiction specific EPO rules. Even so, extra safeguards against accidental or inadvertent use can be and should be implemented risk by reducing unecessary artifacts.

7 People Strategy

8 Find the right people and invest in them. Data center operations is a demanding job and a sector that is growing. While we'd love to find specific data center experience, so does everyone else. We often bring in people from other areas, but we look for trade and situational experiences that are mission-critical in nature. Veterans make great hires. While we don t discriminate against non-military applicants, we certainly encourage them. Why is that? Because they look for success via teamwork. They realize it s about the group outcome, they are happy to follow procedure, and they realize there are consequences if they don t. One of the key things any data center operator can do is put together a team that has the ability to lead and be led. Safety is no accident. Another thing we look for is a history of accident-free work. We ve never had a work injury during construction or operations. If it can t be done safely, we don t do it. Skill Growth. At Infomart, we pay over-market wages and we over-invest in our people, and train them. We offer a substantial amount of training for our staff: 120 hours per year of paid-for training and professional development, safety training and on the job training.

9 Process & Tools

10 Asking the right questions about process. During the RFP process, brokers and potential tenants often resort to an easy question do you have a bunch of engineers who will work in the middle of the night in case there s a problem? It s understandable why the question is being asked maybe back when data centers were unstable you needed a bunch of guys to put their fingers in the hole in the dike, so that question was valid. But the question that SHOULD be asked, and the key to determining the single most important metric measurement in the stability of your organization, is this: how much of your work is PLANNED? In our facilities, that number is between percent. What that means is of all the work orders that get issued, there s a 1 in 50 chance that it was unplanned. What that means is we do work before it has to be done. We don t need the engineers in the middle of the night because those phones don t ring. The alarms don t go off. How does this work?

11 ISO based process strategy. Everything is covered. The moment we buy something, that piece goes into our maintenance management system. Every piece of equipment, every component, every piece of the system. With that comes set points, with that comes setup of preventative maintenance. Everything is transparent. We publish our preventative maintenance plans for the year to all our vendors and customers. First of all, that prevents surprises, but more importantly, it helps keep us from missing anything. Completed work order documentation is shared with tenants and is kept on file for review at any time. Use automated tracking. Preventative maintenance is scheduled monthly and records are reconciled regularly with completion records. Quality control checks. Every successful process has to have feedback. We randomly test 10-15% of our equipment per PM cycle, and perform a monthly reconciliation of non-generated, closed, and open work orders with the engineering team. It doesn t sound like rocket science, but if you don t have all four of these components, your planned work percentage is going to drop.

12 10,000 points of light. At Infomart, we have over 10,000 set points at each facility, and 3,000+ of those are alarmed, giving us a keen amount of insight into all critical systems. We have a very specfic process for investigating alarms that allow us to limit staff required for after-hours work. When an alarm goes off, the first thing we do is verify if it is real alert. Our facility is staffed 24/7/365 and Level 1 response time averages 5 minutes. We only get about alerts a month requiring human attention. More than half of those are cleared due to an blip in communications. For those occasional situations where an alert is validated (less than 1-2 per month), we staff our facilities with engineers who are on-site during regular business hours or on-call nights and weekends. On-call status is a paid position at one quarter of the standard salary; and it requires the engineer to live close to the facility, so that they may respond within 15 minutes or less to situations that require it. 98% of these Level 2 repairs are completed within one shift of alert receipt and account for only 2-3% of total engineering work at our facilites. Employing sev 1 engineers to be on site 24/7/365 would increase costs to the point where our business would be unprofitable. So we have to pick the right tradeoff between reliability and pricepoint. We find having qualified staff on-call during off hours is the right balance: reliability does not come from having staff sit around waiting for the phone to ring, it happens with all the things you did to prevent outages in the first place. Did you have the right design? Did you hire the right kind of people? Did you plan the work? Was the work done consistently? Answering yes to these important questions is the best step you can take to mitigating human error in your facility.

13 But does it work? Infomart s facilities have benefitted from implementing these strategies for Design/People/Process: 300M kw hours of uninterrupted power 100% uptime since inception 0% turnover in engineering staff

14 M&O Certification from the Uptime Institute. In reviewing a site s Management and Operations, Uptime helps organizations analyze the causes and methods to minimizing human error the leading cause of data center downtime. When making an M&O Assessment, Uptime looks at the following facets of effective datacenter management: Staffing and organization (35 points) Training and professional development (20 points) Preventative maintenance program and processes (30 points) Operating conditions and housekeeping (5 points) Planning management and coordination practices (10 points) It should be noted that Staffing and Training the human element make up more than half the total score for effective data center operations. Infomart s Silicon Valley facility was the first recipient of the certification in The facility recertified in 2014 and included the Oregon location as well. Our scores showed improved excellence over two years, and our Silicon Valley facility received a perfect score in both staffing and training categories. With a continued focus on design, and processes, we intend to improve again for the next certification. Any data center, of any tier level, of any age, can benefit from this certificaiton. Making these simple improvements in your data center operations is truly a sustainable way to mitigate human errors and eliminate downtime at your facility. LEARN MORE ABOUT INFOMART S INVOLVEMENT IN M&O CERTIFICATION

15 Silicon Valley Portland Dallas Ashburn infomartdatacenters.com