Preparing to Operate a New Data Center Facility. AFCOM October 3, 2012

Transcription

1 Preparing to Operate a New Data Center Facility AFCOM October 3, 2012

2 Background Managed 100k sq.ft. critical data center facility for GTE Data Services (now Verizon) Site Uptime Network Program Director ComputerSite Engineering team leader David Boston Consulting since 2006 Critical Facilities Operations, including new facility start-up strategy and implementation assistance, procedures and training programs, assessments

3 Proper Planning An enormous amount of planning and coordination goes into the justification, design, and construction of a new data center facility Commissioning Factory acceptance Site acceptance Total systems integration

4 Proper Planning - continued

5 Proper Planning - continued It is equally important to thoroughly develop your strategy for staffing and operating the facility Much of the strategy must be implemented while the building is still under construction

6 Why is This Critical? Facilities related downtime for computer operations is caused by human error 60-80% of the time Processes are just as critical as equipment

7 Why is This Critical? - continued Companies that are serious about uptime dedicate significant effort to Facilities staff plans, procedures, and training, in addition to constructing systems designed for high reliability A data center facility will only operate successfully if the Facilities team is provided management support, appropriate resources, site-specific systems experience, and the resulting confidence level

8 Developing a Strategy Clarify your new facility s reliability objective Match Facilities Operations strategy to systems availability level expected This drives staff size, shift coverage, operating budget It will also dictate the level of training and procedures programs needed

9 Organizational Structure Determine optimal structure to maximize success Which department will Facilities Operations report to in order to receive the best communication, funding, and resources? (And achieve optimal shared processes for hardware master planning and power/network installation)

10 Sample Staff Plan

11 Annual Objectives Facilities staff objectives should include collective goals for consistent facilities systems uptime and successful/safe completion of all assigned PM tasks and customer requests Individual objectives should vary by position, allowing ownership of specific systems, tasks, and projects to be clear (accountability)

12 Hiring Timeline Hire the Facilities team early enough in the design/construction process Involvement in construction monitoring will pay off Several should participate in factory acceptance, site acceptance, and integrated systems commissioning

13 What is the Right Timing? What do you think the appropriate timing is to hire the Facilities Manager? Shift supervisor? Shift engineers? Support services coordinator? Project coordinator? Administrative assistant?

14 Almost Always Overlooked Unless 2-3 months are scheduled after the construction and commissioning process for Facilities training, the Facilities team cannot develop the needed confidence level to successfully respond to systems incidents Computer Operations and Network Operations groups can also use this time period to advantage

15 Procedures Recommendations A comprehensive program is needed ( Facilities procedures, depending on the facility design) A number of the desired procedures should be developed and tested prior to initial operation (tested during the 2-3 month window), if the schedule allows for this

16 Procedures - continued Unless all are completed in advance, a priority order for procedure development is important: 1. Emergency response procedures first 2. Planned system transfer procedures in the order that equipment PMs will be scheduled 3. MOPs for daily PM tasks (less risk)

17 Procedures - continued One of the Facilities team members should be assigned the responsibility of Procedures and Training Coordinator The Coordinator must be given resources to accomplish this In-house staff assigned to help part time Contracted system experts and writer

18 Procedures - continued Ensure that all procedures are tested What is clear to the author often leaves room for confusion for the reader If unclear, the procedure may actually introduce risk

19 Procedures - continued It is critical that two individuals participate in each planned activity that utilizes a procedure One to read aloud, the other to confirm and act This greatly diminishes the risk of skipping a step or performing a step out of order It also encourages discussion and a potential decision to back out if there is any confusion

20 Sample Procedures Format 1. Find breaker control switch GTA HVEA GEN. TIE BREAKER in the bottom row of controls in Master Control Cubicle MC 2. Turn this control switch counterclockwise to the TRIP position and then release

21 Training Program Each equipment vendor will typically deliver training after equipment has been installed Scheduling should be managed to prevent overload Review each vendor s materials in advance Professionally video record the training

22 Training Program More critical than vendor training is the implementation of monthly site-specific training (train on each system at least annually) January: Generators, controls, & fuel systems February: Life safety and evacuation March: Cooling towers, chillers, pumps, valves April: UPS, UPS switchgear, batteries May: CRAHs, VAVs, AHUs Etc.

23 Training Program - continued The Procedures and Training Coordinator should involve many of the identified system experts to conduct initial monthly sessions In-house staff should become trainers over a planned timeline Classroom (group) and in-front-of-equipment (individual) sessions each month

24 Training Program - continued Engineers of Record or Commissioning Agents should be contracted to deliver Systems Design Concepts and Systems Overview training

25 Training Program - continued Identify system experts to help with initial inhouse monthly training sessions

26 Training Program - continued Once your site specific program is established: Internal certification Recognition Planned progression

27 Budget Planning Experience provides the best resource for planning your new facility s expense budget If your company has operated a data center previously, that expense history should be utilized when developing the new facility s initial budget History within same region most useful

28 Budget Planning - continued Your electrical engineer of record and your electric utility representative can help you estimate your initial year s utility costs Utility expenses are consistently the largest operating expense in a data center facility

29 Budget Planning - continued The second highest cost category will be labor and related expenses for Facilities Operations staff (in-house or contracted)

30 Budget Planning - continued Collectively, maintenance contracts for preventive maintenance (PM) on major systems will typically be the next highest expense Determine level of PM conducted by staff vs. vendors

31 Budget Planning - continued Remaining categories include: Spare parts, materials, tools, uniforms, supplies (First confirm what construction budget will cover) Janitorial (including sub-floor cleaning) Grounds Security (if Facilities is responsible)

32 Budget Planning - continued Expenses for individual data center facilities vary widely due to: Cost of electric utility service (kwh) Levels of redundancy & concurrent maintainability Load variances: types of computer hardware installed Salary & benefits for Facilities Operations team Union vs. non-union staff Single shift vs. continuous shift coverage Number of employees per shift Outsourced vs. in-house staff

33 Orientation Program Formal playbook to accelerate knowledge & confidence acquisition for new team members Include: Introduction (why site critical, cautionary philosophy) Systems overviews and descriptions One line diagrams, building layout Safety and Security policies Training session and major PM calendar, first year expectancies Prescribed individual shadow program

34 Workspace Strategies Benchmark with peers or hire service providers to help you maximize functionality for: Equipment rooms, parts storage areas Facilities shop, office, command center/library Loading dock, unpacking, staging and burn-in areas

35 Additional Strategies for Success Control processes Specific data center work rules Limited access minimize those permitted badges Shipping/receiving only on a planned basis Computer hardware installation planned in advance

36 Additional Strategies - continued Team Development Clearly defined IT and Facilities relationship, mutual expectations (or SLAs), shared incentives Defined Data Center Facilities and Office Facilities relationship (if other buildings on campus)

37 Measuring Success Effectively deploying these strategies will provide for a much higher reliability potential in the first year of operation Continued application will dramatically increase continuous facilities systems operation potential over the life of the facility

38 Measuring Success - continued You may realistically aim for multiple years of continuous facilities systems availability, a multi-million $ savings

39 Questions?

40 Web site: Contact Information References: upon request Phone: Fax: Address: 215 Wildwood Dr. Highlands, NC 28741