You Manage What You Measure

Transcription

1 You Manage What You Measure June 17, 2008 AUTHOR: Ken Anderson TECHNOLOGY THREAD: Management

2 Synopsis IT departments rely on numerous operational metrics to accurately set goals, meet departmental expectations, and improve overall productivity. IT managers reinforce the importance of metrics with the oft-quoted IT mantra you can't manage what you don t measure. Metrics also affect organizational behavior because IT employees focus on meeting expectations defined by their individual metrics. While IT managers utilize common frameworks to define, combine, and assign weight to important metrics and place them into manageable groups, IT employees understandably focus on singular metrics that directly relate to their job performance. The net result is that you manage what you measure. Analysis Metrics are a critical element of any successful business. High-level business metrics define impacts and outcomes such as market growth, marketing efficiency, customer trends, and product profitability. In comparison, IT metrics tend to be internal, focusing on operational efficiency and demonstrating value for money spent. When corporations implement new technology, IT departments define new metrics to better manage the increasingly complex environment. As IT metrics grow in number and complexity, they require the deployment of sophisticated management systems to collect and display operational data. As IT departments struggle in today s more businessoriented, less technical landscape, IT management is starting to realize that traditional IT metrics can t articulate the true value of IT. Many IT departments are utilizing common frameworks to customize, combine, and assign weight to important metrics and place them into manageable groups. Some IT departments are combining various metrics into executive dashboards while others are focusing on key metrics as indicators of progress. IT employees understandably focus on singular metrics that directly relate to their job performance. Comfortable with their traditional metrics, they struggle as IT management makes changes. Although executive dashboards give upper management a concise view of IT, they do little for individual employees trying to prove their value. In the end, as corporations scrutinize the business value of their IT investments, IT departments struggle to show their value though measurable results. IT departments all too often focus their energies on defining, managing, and publicizing numerous metrics that are void of any business expectations or understanding of what the business needs to have measured. While IT employees ask for simplified metrics to prove their value, corporate executives are asking for the same to understand IT s business value. A Metric is a Metric by Any Other Name: What Is a Metric? A simplistic definition for a metric is a standard of measurement. The key terms are standard, which implies agreement between parties, and measurement, which implies an understanding of value. In addition, each metric s individual characteristics play an important role in understanding its business impact. Metrics: Absolute vs. Relative IT metrics, such as outages or uptime, are expressed in absolute terms because we compare the result to the absolute best result of 0 or 100%. While perfection is possible, it may never be realistic because the cost of being perfect may not be justified by the benefit. Although IT professionals take great satisfaction from expressing their uptime in percentages, real value lies in understanding actual requirements and expressing metrics relative to business needs. Providing five nines (99.999%) uptime

3 might be impressive to other IT departments, but a CFO may not be thrilled with the cost if the business tolerates less than perfection. Understanding the distinction between absolute and relative metrics is useful when communicating to a business audience. By presenting the absolute metric and comparing it relative to industry peers, the metric is presented with context. This context allows the business to better understand what kind of investment may be required and to eventually make a more informed decision. Metrics: Efficiency vs. Effectiveness IT departments rely on efficiency metrics to communicate status and success. Efficiency metrics are easier to measure because they are derived from known data sources and generally presented as factual. Common IT efficiency metrics include capacity, project progress, bugs logged, and expense against budget. Effectiveness metrics are more difficult to implement because they define impacts and outcomes; data points require analysis and outputs are subject to interpretation. These metrics tend to be business-focused and measure tangibles such as market growth, marketing efficiency, customer trends, and product profitability. IT departments that are skilled in efficiency measurement experience a disconnect when their business customers expect more effectiveness metrics. To solve this problem, IT departments require their business customers to translate business needs into detailed requirements documents that can be more efficiently managed. This process often strips the business context requirements which define effectiveness out of the system definition. Without the corresponding business detail, IT departments may complete projects on time and within budget without meeting the more subtle requirements for effectiveness. The Gravity of Metrics IT employees understandably focus on singular metrics that directly relate to job performance. The problem is that each metric acts like a planetary body with its own gravity; as the metric moves, so do other metrics. This effect can cause waves of confusion, and while priority metrics are met, overall performance deteriorates. High and Low Tide at the Helpdesk The financial justification of a helpdesk or call center is based on the concept of entry level technicians solving incidents previously handled by more expensive personnel. A key driver for success is the percentage of incidents resolved by the helpdesk in comparison to those requiring escalation. The cost per incident resolved compared to traditional support costs is a common helpdesk fiscal metric. IT managers trying to lower their cost per incident resolved will turn to common IT metrics such as call completion rate or number of calls solved per technician. The reasoning is sound: If the helpdesk can solve more calls, (while escalating fewer calls) or if each technician completes more calls per day, then the cost per incident resolved will be lower. The Team Achiever The IT helpdesk technician with incentive to close a higher percentage of incidents will spend more time on the phone, making every effort to resolve the issue before escalating. Because the technician spends more time on the phone, customers wait longer before the call is escalated and ultimately solved by a more experienced technician. As this behavior is replicated throughout the helpdesk, fewer technicians are available to answer new requests. Goal Result Increase efficiency by increasing number of calls solved before escalation Increased average time per call, wait time before answer, and calls abandoned The Individual Achiever The IT technician with incentive to close more calls solved per technician will spend less time on the

4 phone, solving the easy incidents first and quickly escalating more complicated calls. Goal Result Increase efficiency by increasing number of calls solved per technician Decreased number of calls solved before escalation and average time per call Blending Metrics: Opposites Attract An effective measure for reducing the gravitational pull of a particular metric is to find its equal-and-opposite metric. In the helpdesk example, the call completion rate metric motivated technicians to spend more time with a customer before escalating to the next tier of technical staff. Combining call completion rate with the duration per call metric creates a paired metric that provides balance. The grouping must carry its own performance values; technical personnel cannot only be educated on the importance of the paired metrics. An example of a paired Metric would be helpdesk calls closed within five minutes. The Molecular Structure of a Metric IT departments are finding ways to simplify traditional IT metrics into a more consumable executive-friendly format. Reverse engineering a metric to understand its basic structure is also advantageous. Although this may initially seem counterintuitive, the result can provide a more complete understanding of what drives a particular metric. System availability or uptime is one of the most recognized metrics from an IT department. System availability is an efficiency metric generated by subtracting downtime from total possible availability. Therefore, the more downtime one has, the less available the system is. By reverse engineering downtime, we can break it into three components: Notification Escalation Resolution The elapsed time before IT is notified of an outage. The elapsed time required to find the correct person to resolve the issue The elapsed time required to resolve the issue Understanding the components of downtime helps IT departments understand how to reduce downtime and increase system availability. IT departments routinely focus on the resolution when the actual need is to focus on better notification and escalation. Conclusion The oft-quoted IT mantra you can't manage what you don't measure, is as applicable today as it was a decade ago. The idea of running a business without measurable outcomes is a good definition for enterprise anarchy. Although the concept of metrics is a sound business practice, not understanding how to use metrics can also result in similar forms of anarchy. Ultimately, by defining the measurements that will be managed, you will be managing what you measure.

5 Related Burton Group Research Executive Advisory Program IT: The Business of Technology A CIO s View of Server Virtualization A New Classicism What Does a CIO Actually Do? Executive Blogs: A Survey Network and Telecom Strategies Enterprise SLAs: Lessons Learned Key Performance Indicators for Enterprise Networks Copyright 2008 Burton Group. ISSN All rights reserved. All product, technology and service names are trademarks or service marks of their respective owners. See Terms of Use and publishing information at