How To Manage A High Risk It Event

Transcription

1 Jump-Starting the Centralizing Communication for High-Risk Information Technology Events

2 Introduction Communication among key internal teams and quick, organized incident resolution for customers are essential to preserving business continuity when organizations undertake high-risk and high-impact information technology (IT) events. A command center, with a well-trained team devoted to documenting, prioritizing, escalating and helping to resolve incidents, can help ensure the success of such a major IT event and reduce the risk of business interruption. The command center is not a project management organization, but a temporary cross-functional team composed of the appropriately skilled personnel organized in advance of planned events and designed to serve potentially as a 24/7 main line of support during pre-production and post-go-live IT events that are mediumor high-risk and/or high-impact to internal and external operations and customer transactions. A large-scale data migration or conversion event where major applications are being implemented or modified would be a likely candidate for a command center, but an organization may implement the function to support other IT events such as a pilot, acceptance testing, performance/scalability testing, mock testing or a production release. 1

3 Basic Structure Senior Leader Makes key decisions Assigns severity levels to incidents Escalates issues when necessary Coordinator Ensures staffing plan Secures necessary resources Establishes documentation and processes Incident Application SMEs Infrastructure Representative Resolution Administrative role to record all issues and decisions Analyzes incident reports for trending issues Provides roll-up reporting Provides insight into detailed operations Resource to resolve key issues Provides general oversight of release Provides expertise in infrastructure components Facilitates resolution of issues that are reported to the command center Coordinates the paging of necessary on-call and support resources Figure 1: Basic structure for a command center (smaller IT events may require fewer staff members) Establishing the : A Flexible Structure A command center s staffing level and duration will vary based on the level of risk and need for dedicated support for a specific high-risk IT event. The organizational chart above outlines a suggested structure for a command center and details the roles of personnel typically required. The command center s senior leader is the key decision-maker who assigns severity levels to reported incidents and escalates issues when necessary. The severity level, which is assigned before an incident is passed on to a subject-matter expert (SME) for resolution, dictates how closely and quickly the issue must be monitored or resolved. This role is typically filled by a senior leader from the client s existing structure who can act as a liaison between the command center team and other executive leaders within the organization. The command center coordinator, who reports to the senior leader, ensures the staffing plan, secures the necessary resources (e.g., rooms, computers, bridge lines), and establishes documentation and processes related to incident reporting around the IT event. The incident manager interacts directly with customers who contact the command center, gathering all pertinent information about an issue and then passing it on to the relevant support groups. The application SMEs (resources with the knowledge and insight to resolve key issues); the infrastructure representative (who provides expertise into infrastructure components, such as server, network and database components); and the resolution manager (who facilitates resolution of reported issues) round out the team. Smaller IT events (such as minor code deployments, testing events or other lower-risk activities) may require only one or two staff members to perform the responsibilities of each role in the basic structure. Additionally, the incident manager and resolution manager roles may be combined. The command center organization is meant to be flexible and assign roles as necessary to support the type and significance of the event as needed. 2

4 The War Room During high-risk IT events that typically require elevated staffing levels and rapid response, it may be necessary for an organization to separate staff between a command center and a war room function. A war room is a specialized team of SMEs drawn from across the business who are tasked with resolving high-priority incidents as they occur, freeing the command center to triage and route less critical tasks to other support groups. The command center staff (see blue boxes in Figure 2) handles incoming issues and assigns incidents to the war room staff (see gray boxes in Figure 2) for investigation and resolution. Senior Leader Makes key decisions Reviews severity levels assigned to incidents Escalates issues when necessary Coordinator Ensures staffing plan Manages command center operations during the event Responsible for communications updates to other command centers when necessary Sends status communications (deployment/incident) as agreed upon with each team Incident Application SMEs Infrastructure Representative War Room Lead Administrative role to record all issues and decisions Analyzes incident reports for trending issues Assists coordinator in providing roll-up reporting Provides insight into detailed operations Resource to resolve key issues Provides general oversight of release Provides expertise in infrastructure components Facilitates resolution of issues that are reported to the command center Contacts enterprise support groups when necessary Coordinates the paging of necessary on-call and support resources Figure 2: Some high-risk IT events may require a separate war room staff 3

5 Sending the Right Information to the Right People The command center helps promote effective communication about incident resolution among support personnel, project teams, other support teams and command centers, and IT/application groups. It receives initial calls for help from internal customers (i.e., the application, test or pilot teams) when issues, such as a service outage, arise during a major IT event. Depending on the size and duration of the event, customers may be given several ways to contact the command center for example, a toll-free number, a conference bridge line, an instant messaging application, or a shared box. They also may be able to submit their issues electronically through a designated support portal. The command center team documents and then relays information to the appropriate SMEs for resolution. The team logs all calls and s it receives from customers, whether the issue reported is an incident (e.g., degradation of service) or a defect (i.e., a bug or issue discovered as a result of testing), and provides level one support for the caller. Additionally, the command center will interface with other IT Service Management groups throughout the enterprise and ensure that appropriate resolution teams are engaged. While the command center does not function as a centralized defect reporting center, or fix defects, it will log such reports into the tracking system and communicate that information to the testing team, which establishes the process for handling reported defects. At the close of each supported IT event, statistics related to the reported issues are calculated and inserted into a report that is presented to project managers, program leads and work stream leads. The information the team documents about reported errors and defects and how they were resolved is useful knowledge the organization can apply in the future when preparing to undertake and provide appropriate support for other significant IT events. Conclusion When establishing a command center, organizations should strive to create a structure that will be easy to replicate quickly and lessons learned after each event is completed should be applied as soon as possible. And when undertaking high-risk IT events, they also should identify a set of metrics for measuring results that are both meaningful to management and usable by support staff. Primary metrics should be in alignment with an organization s business as usual status, and be actionable. If failures, missed targets, or data outliers are discovered, appropriate and timely action can be taken to investigate and remediate such failures. Technology is always changing, and so too are business needs, which means the likelihood of a large organization undertaking more than one high-risk IT event in the near future is likely. The support that a designated command center function can provide is necessary for the success of any high-risk IT project; there is simply too much at stake for an organization not to have a well-trained team in place to manage centrally the incident reporting and resolution process throughout all critical stages of a major event. 4

6 About Protiviti Protiviti ( is a global consulting firm that helps companies solve problems in finance, technology, operations, governance, risk and internal audit. Through our network of more than 70 offices in over 20 countries, we have served more than 35 percent of FORTUNE 1000 and Global 500 companies. We also work with smaller, growing companies, including those looking to go public, as well as with government agencies. Protiviti is a wholly owned subsidiary of Robert Half International Inc. (NYSE: RHI). Founded in 1948, Robert Half International is a member of the S&P 500 index. The in Action While assisting a major financial institution with merger activity, Protiviti s on-site project teams identified a need to mitigate a number of production support-related risks. A key middleware team implementing a new platform to eliminate a significant number of partner-to-partner interfaces was concerned about how to control the inherent risks associated with upcoming deployment events. Making these changes was critical to allow legacy and go-forward systems from both organizations to communicate effectively without extensive code changes. Protiviti worked with the client to develop and implement a command center and ongoing production support model. In its final form, the command center was a joint effort between our team and the client s staff. Protiviti created much of the organizational structure and the leadership team, and was responsible for back-office preparations both before and during the supported events. The core team was augmented with SMEs from application teams during high-risk events, who provided the necessary details as production incidents occurred. This model allowed the command center to successfully integrate into the overall corporate command center structure. About Our IT Consulting Protiviti s IT Consulting practice helps IT leaders design and implement IT governance, security, data management, application and compliance solutions. Our professionals partner with CIOs and other IT and business leaders to ensure their organizations maximize the value of information systems investments while mitigating key risks. Our IT Consulting practice has brought technology and business solutions to more than 1,200 clients worldwide. We group our IT consulting services into three areas of focus for CIOs: Managing the Business of IT Managing IT Security & Privacy Managing Applications & Data We invite you to explore at how we create value by helping clients identify and solve critical business and technology issues. For additional information about the issues reviewed in this white paper or Protiviti s services, please contact: Michael Schultz Managing Director [email protected] Jonathan Wyatt Managing Director [email protected] Jason Riddle Senior [email protected] Kyle Nesgood [email protected] 5

7 The Americas Europe United States Alexandria Atlanta Baltimore Boston Charlotte Chicago Cincinnati Cleveland Dallas Denver Fort Lauderdale Houston Kansas City Los Angeles Milwaukee Minneapolis New York Orlando Philadelphia Phoenix Pittsburgh Portland Richmond Sacramento Salt Lake City San Francisco San Jose Seattle Stamford St. Louis Tampa Washington, D.C. Woodbridge France Paris Germany Frankfurt Munich Italy Milan Rome Turin Middle East The Netherlands Amsterdam United Kingdom London Argentina Buenos Aires* Brazil Rio de Janeiro* São Paulo* Canada Kitchener-Waterloo Toronto Mexico Mexico City* Monterrey* Peru Lima* Venezuela Caracas* Bahrain Manama* Kuwait Kuwait City* Oman Muscat* United Arab Emirates Abu Dhabi* Dubai* Asia-Pacific Australia Brisbane Canberra Melbourne Perth Sydney China Beijing Hong Kong Shanghai Shenzhen India Bangalore Mumbai New Delhi Indonesia Jakarta** Japan Osaka Tokyo Singapore Singapore South Korea Seoul * Protiviti Member Firm ** Protiviti Alliance Member 2012 Protiviti Inc. An Equal Opportunity Employer. PRO Protiviti is not licensed or registered as a public accounting firm and does not issue opinions on financial statements or offer attestation services.