10 Evaluating the Help Desk

Transcription

1 10 Evalating the Help Desk The tre measre of any society is not what it knows bt what it does with what it knows. Warren Bennis Key Findings Help desk metrics having to do with demand and with problem resoltion time seem most interesting to or respondents; telephone-related metrics seem less so. Central IT staff and management are the primary constitencies to which help desk metrics are reglarly reported. Or respondents feel they do fairly well in sing metrics to improve ser service. This is especially tre among those who report their metrics widely and in a variety of ways, and among those who se basic IT service management practices. Informal methods of assessing ser satisfaction are more common than formal ones, althogh formal srveys and Web feedback forms are also common. Respondents generally agree that the vale of the help desk is well docmented and nderstood. Agreement varies abot whether help desk costs are being well docmented, bt a majority disagree that costs are well nderstood. Help desk matrity was positively associated with the nmber of or for basic IT service management practices for which the IT organization had formal gidelines in place and the perceived adeqacy of the help desk s involvement in them. Matrity was also positively associated with more robst help desk toolsets for help desk administrators, staff, and clients; the nmber of goals the help desk has adopted; the stats of strategic planning for the help desk; and the alignment of camps expectations of the help desk with its resorces. Other positive associations with help desk matrity were the nmber of help desk metrics reglarly analyzed, the mean nmber of methods sed to assess ser satisfaction, and agreement that help desk costs and vale are well docmented and well nderstood. A set of reglarly measred performance indicators is an important tool in aligning central IT help desk performance with the needs of the constitencies it serves. To gain a sefl perspective on the help desk, its managers mst monitor progress toward goals that are focsed internally as well as others with a more external focs. As Ed Pittarelli, director of technologies at Bergen Commnity College, pts it, Withot 2007 EDUCAUSE. Reprodction by permission only. EDUCAUSE Center for Applied Research 99

2 metrics, no one scceeds. If yo let the operation rn free-form, it gravitates to the lowest possible level of service. Obviosly, it isn t enogh to merely collect metrics; the help desk mst pt that information to work within the help desk and the central IT organization, and share it with varios constitencies to inform the camps of both the costs and the vale of its services. In this chapter, we look in depth at the choices or respondents have made in these areas. As we will see, gathering and collecting performance information, as well as the preliminary steps of strategic planning and goal setting, are matre processes, as defined by a leading process matrity model. While achieving process matrity may not be a stated goal of the help desk, we will discover that sch a goal embraces many of the more specific goals or respondents reported prsing in Chapter 9. Basic Help Desk Metrics Help desk metrics inclde sch inpts as the nmber of clients and devices spported or the nmber of spport reqests received per staff member. They also inclde otpts, sch as the percentage of problems resolved dring the client s initial contact with the help desk or the average time it takes to resolve problems that can t be resolved dring that initial contact. Beyond simple inpts and otpts, they also inclde service qality measres sch as the average time a caller mst wait before being served or the rate at which callers on hold abandon their attempts to get spport. Gathering the Data Not all help desks collect the same metrics, and the freqency with which they analyze them varies as well. Becase freqency of analysis is the better indicator of a metric s effectiveness, we chose to ask which of eight metrics central IT help desk personnel analyzed, and with what freqency (see Figre 10-1). Of the eight help desk metrics we asked abot, the help desk staff reglarly (at least once a year) analyzed only two at majorities of or respondent instittions. These were call/contact load and nmber of sers spported. Roghly an additional qarter of respondents analyzed each of these two metrics on an ad hoc basis, indicating that 87 to 88 percent of respondents make some se of that information. Abot 45 percent of respondents reported reglar analysis of three of or metrics: the nmber of problems resolved at first contact, the time it takes to resolve a problem that cannot be resolved at first contact, and the nmber of devices the help desk spports. An additional 25 to 30 percent of respondents analyzed these metrics ad hoc, indicating that abot three-qarters of respondents make some se of that information. At majorities of respondent instittions, help desk staff did not analyze two metrics related to telephone clients, nor did they analyze the nmber of help desk contacts (calls) per device. The telephone-related metrics inclded the length of time a client had to wait on the telephone before getting a response to a problem and the rate at which clients hng p the telephone rather than wait for a help desk staff member to answer. In the case of telephone wait times, not qite a third of respondents analyzed that metric reglarly, and another 12.0 percent analyzed it ad hoc. Slightly more than a qarter of respondents reglarly analyzed call abandonment rate, and slightly fewer than half that many analyzed it ad hoc. Of the eight metrics we chose to stdy, the nmber of contacts per device was reglarly analyzed by the fewest respondents: jst 2 in 10. However, almost a qarter analyzed this metric ad hoc, so nearly half of respondents analyzed it at least sometimes. 100

3 Call/contact load (N = 452) Nmber of sers spported (N = 452) Problems resolved at first contact (N = 451) Nmber of devices spported (N = 451) Problem resoltion time (N = 451) Figre Freqency of Analysis of Selected Metrics Telephone wait times (N = 451) Call abandonment rate (N = 451) Contacts per device (N = 451) % 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Not analyzed Analyzed ad hoc Analyzed qarterly or annally Analyzed weekly or monthly Analyzed daily or continosly Percentage of Instittions Almost half of respondents (49.1 percent) analyzed six or more of the metrics we asked abot. Nearly 4 in 10 (39.2 percent) analyzed three to five of them, and 6.8 percent analyzed jst one or two. Only 4.8 percent of respondents reported that help desk staff analyzed none of the metrics we asked abot. The nmber of help desk metrics analyzed varies significantly by both Carnegie class and FTE enrollments. Or findings sggest that larger and more academically complex instittions find it more desirable or necessary to formally track help desk performance than instittions that are smaller or offer fewer academic options. Sharing the Data In most cases, information abot the central IT help desk mst be shared if it is to be sed effectively. Or respondents told s that their help desk metrics were shared mostly within the central IT organization and that a variety of vehicles were sed to share it. As Figre 10-2 indicates, central IT staff and management, inclding the CIO, were the primary constitencies to which help desk metrics were reglarly reported, each being cited by two-thirds of respondents or more. Only abot half as many respondents said they reglarly report help desk metrics to senior administrators (president or chancellor, vice presidents, and cabinet-level officers). Less freqently cited as reglar recipients of help desk metrics were deans, non-it management, faclty, non-it staff, and stdents. We asked how respondents reported metrics to their varios constitencies and offered as choices six standard vehicles, as well as Other (see Figre 10-3). At 49.9 percent, the most freqently cited vehicle was Other, sggesting that or list of six missed at least one important option. While EDUCAUSE Center for Applied Research 101

4 Central IT management (N = 445) 81.8 CIO (N = 441) 78.5 Central IT staff (N = 444) 68.7 Figre Constitencies to Which Help Desk Reports Metrics Senior administrators (N = 429) Deans (N = 427) Non-IT management (N = 432) Faclty (N = 423) 12.1 Non-IT staff (N = 432) 10.6 Stdents (N = 434) 7.6 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Percentage of Instittions Other (N = 381) 49.9 IT organization annal report (N = 433) 44.3 Figre IT newsletter (N = 428) 21.0 Vehicles by Which Help Desk Reports Metrics to Constitencies Help desk Web site (N = 433) Exective dashboard (N = 431) Other Web sites (N = 432) 7.4 Camps portal (N = 429) 5.8 0% 10% 20% 30% 40% 50% 60% Percentage of Instittions we did not ask respondents to specify which other vehicles they sed, we speclate that many respondents wold have told s they se formal and informal written reports to commnicate metrics to some of the constitencies discssed above. Effective Use of Metrics When asked to respond to the statement that the help desk ses metrics effectively to improve ser service, nearly a third of respondents agreed and 8.7 percent strongly agreed (see Figre 10-4). Netral responses made 102

5 Strongly agree, 8.7% Strongly disagree, 8.5% Agree, 32.3% Disagree, 21.8% Figre Agreement That Help Desk Uses Metrics Effectively (N = 449) Netral, 28.7% p somewhat less than a third of responses, while negative responses totaled nearly a third. As we will see later in this chapter, the effective se of metrics to improve ser service was positively associated with help desk matrity; in Chapter 11 we will see that it was also positively associated with the overall qality of help desk services. Agreement that the help desk ses metrics effectively also varied by Carnegie class. Doctorals reported a significantly higher mean level of agreement (3.62, standard deviation 1.010) than other classes. Master s instittions had the lowest level of agreement, at 2.79 (standard deviation 1.042). Bachelor s and associate s instittions were nearly tied at means of 3.05 (standard deviation 1.044) and 3.13 (standard deviation 1.023), respectively. Mean agreement at Canadian instittions was 3.04 (standard deviation 1.113). Mean agreement that metrics are sed effectively is positively associated with several practices we have discssed so far, inclding the nmber of metrics help desk staff analyzed, the nmber of camps constitencies metrics were reported to, the nmber of vehicles sed to report metrics, the stats of service level agreement (SLA) se, the mean nmber of docmented goals in place for the help desk, and the nmber of basic ITSM practices for which the central IT organization had formal gidelines in place. Assessing Satisfaction Perhaps the most important metric a help desk can have is the level of satisfaction help desk sers feel with the services the help desk provides. This offers a window into its own effectiveness. Accordingly, we asked abot this metric in more detail than those discssed above. Or respondents reported sing a wide range of methods to assess ser satisfaction. Figre 10-5 shows that the most commonly sed method was gathering nsolicited inpt from help desk sers. Nine in 10 respondents (90.1 percent) receive sch inpt. Three-qarters of respondents se the more interactive method of informal meetings with help desk sers. Neither of these methods necessarily involves docmentation, and both are likely to yield qalitative rather than qantitative information. Majorities of or respondents sed two considerably more strctred methods: EDUCAUSE Center for Applied Research 103

6 Unsolicited inpt from sers (N = 446) 90.1 Informal meetings with sers (N = 443) 75.4 Formal srveys (N = 447) 62.4 Figre Methods Used by Help Desk to Assess User Satisfaction Web-based feedback forms (N = 440) Formal meetings with key sers (N = 435) Sggestion boxes (N = 436) Point-of-service forms (N = 436) 23.4 Formal focs grops (N = 435) 17.9 Stdies by external consltants (N = 433) % 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Percentage of Instittions Nearly two-thirds condcted formal srveys of help desk sers, and more than half sed Web-based feedback forms. Almost half of respondents reported formal meetings with key sers, presmably more strctred than informal meetings bt still yielding information that is more qalitative than qantitative. Methods reported by fewer than a qarter of respondents were sggestion boxes, pointof-service forms, and formal focs grops. Only 1 in 8 respondent instittions reported bringing external consltants to camps to assess ser satisfaction. Another option available to IT organizations is to contract for an externally based cstomer satisfaction service sch as that offered by HDI (formerly known as the Help Desk Institte), a membership organization for the service and spport indstry. Timothy Farnham, CIO at Berry College, has sed HDI s cstomer satisfaction service for abot a year. It s a pretty good feedback mechanism, he says, and it shold be particlarly sefl in determining sers assessment of the operational changes we have in mind for or help desk. Or srvey findings sggest that, overall, respondents have fairly good qantities of information concerning ser satisfaction. Mch of it is informal and qalitative, bt sbstantially more than half of respondent instittions are able to validate that information against qantitative information from formal srveys or Web-based feedback forms. Commnicating Costs and Vale Figre 10-6 shows that a majority of respondents (53.7 percent) agree or strongly agree that the vale of help desk services is well nderstood, and a near-majority (45.2 percent) agrees or strongly agrees that vale is well docmented. Agreement with or statement that costs are well docmented was distribted bimodally, with a majority expressing positive responses; netral opinions made p abot a sixth of the responses, and negative 104

7 60% 53.1 Percentage of Instittions 50% 40% 30% 20% 10% Figre Level of Agreement That Help Desk Costs and Vale Are Well Docmented and Well Understood 0% Costs are well docmented (N = 447) Costs are well nderstood (N = 439) Vale is well docmented (N = 447) Vale is well nderstood (N = 445) Strongly disagree Disagree Netral Agree Strongly agree responses came from not qite one-third of respondents. However, agreement that costs are well nderstood was overwhelmingly negative, with almost two-thirds of respondents disagreeing or strongly disagreeing and only one in eight agreeing or strongly agreeing. The mean levels of agreement presented in Table 10-1 clearly illstrate the discrepancy between the effectiveness of help desk commnication abot costs and abot vale. Mean agreement abot the docmentation of costs, at 3.36 on a five-point scale, came in slightly more positive than mean agreement abot the docmentation of vale, at However, mean agreement abot camps nderstanding of costs, at 2.39, was more than a fll point below mean agreement abot nderstanding of vale (3.43). Ths, despite slightly greater effort to make costs known, overall or respondents felt camps nderstanding of costs to be inadeqate. Camps nderstanding of the help desk s costs and vale is positively associated with the alignment of camps expectations and help desk resorces, discssed in Chapter 6. Respondents agreeing or strongly agreeing that vale is well nderstood expressed a mean level of agreement that expectations were aligned with resorces that was abot 0.7 points higher, on or five-point scale, than for those disagreeing or strongly disagreeing that vale is well nderstood. The pattern and level of difference were very similar for help desk costs. This finding reinforces or sense that sccessfl commnication of help desk costs and vale can contribte to the alignment of expectations and resorces. Help Desk Matrity To get a feel for the level of development of respondents help desks, we adapted for or prposes the Capability Matrity Model Integration (CMMI) framework developed by the Carnegie Mellon Software Engineering Institte. 1 The framework defines five levels of matrity that can be applied to processes like those ndertaken by a central IT help desk. Those levels, as we adapted them and defined them for or respondents, are as follows: EDUCAUSE Center for Applied Research 105

8 Table Agreement That Help Desk Costs and Vale Are Well Docmented and Understood Docmentation and Understanding N Mean* Std. Deviation Costs are well docmented Costs are well nderstood by constitents Vale is well docmented Vale is well nderstood by constitents *Scale: 1 = strongly disagree, 2 = disagree, 3 = netral, 4 = agree, 5 = strongly agree Initial Services are sally provided ad hoc and rely on individal efforts, and past sccesses are often not repeatable. Repeatable Service responsibilities are formally assigned, sccess is sally repeatable, and basic project management techniqes are sed. Standardized Service qality standards are in place and sed, consistency of services is a priority, and process improvement is a goal. Managed Qantitative performance goals are in place, service performance is measred, and service qality is predictable. Optimized Services are closely aligned to bsiness strategies, services are easily changed to meet emerging needs, and process improvement is continos. In the following discssion, we refer to the categories nearer to optimized as more matre and those nearer to initial as less matre. Figre 10-7 portrays the range of assessments or respondents made of their help desks matrity. Slightly more than one-third of respondents (35.7 percent) reported the lower initial and repeatable matrity levels. A slightly larger grop, 40.0 percent, reported the middle level of matrity, standardized, and not qite one-qarter (24.3 percent) reported the more matre managed and optimized levels. We fond no significant association between help desk matrity level and any of or key demographics, the central IT organiza- tion s goal, the IT bdget, instittional bdget climate, or help desk staffing, all of which we might expect to affect the help desk s performance along the dimensions inclded in or matrity framework. Nevertheless, many other srvey items were positively associated with help desk matrity, as discssed in the following sections. The Matre Help Desk Toolset Among the factors associated with help desk matrity, as reported to s, was the mean nmber of tools the help desk sed and provided for its staff and its clients. As reported in Chapter 7, we asked abot the atomation of five common help desk fnctions: call logging, call roting, call escalation, a call database, and call database qery and reporting tools. Table 10-2 shows that the mean nmber of fnctions for which atomation was flly implemented was smaller for instittions reporting less matre help desks and higher for those reporting more matre ones. Similarly, respondent instittions with more matre help desks reported the implementation of significantly higher nmbers of help desk staff and ser spport tools, and greater agreement that the instittion effectively ses self-service tools to redce help desk demand. Help Desk Matrity, Goals, and Planning As Table 10-3 shows, the nmber of reported goals was associated significantly with perceived help desk matrity p 106

9 Optimized, 6.6% Initial, 7.7% Managed, 17.7% Repeatable, 28.0% Figre Matrity Level of Help Desk (N = 453) Standardized, 40.0% Table Nmber of Atomated Help Desk Fnctions, by Help Desk Matrity Level Matrity Level N Mean Std. Deviation Initial Repeatable Standardized Managed Optimized Total to a point. From initial matrity level to managed, the mean nmber of goals reported increased from 2.97 to The mean dropped off slightly to 6.60 for the optimized category, bt this may simply be an artifact of sample size, which drops sbstantially in this category. Or respondents also gave s reason to believe that strategic planning is positively associated with help desk matrity. Figre 10-8 shows that instittions with no strategic plan for the help desk made p slightly more than half of those whose help desks were at the initial and repeatable matrity levels. Instittions with help desk strategic plans integrated into their central IT organizations strategic plans for IT made p almost 6 in 10 of those with managed or optimized help desks. In general, the better developed and more integrated the instittion s strategic plan, the higher its help desk s matrity level. Having a stand-alone strategic plan for the help desk seems less related to help desk matrity. The percentage of optimized help desks with sch plans was 6.7, a bit more than twice as many as at the initial level. By contrast, the percentage of optimized help desks with integrated strategic plans was more than 20 times the percentage at the initial level. This spports a key tenet of the IT service management literatre that the best environment for IT services is one in which the service providers efforts are well integrated. EDUCAUSE Center for Applied Research 107

10 Table Nmber of Docmented Help Desk Goals, by Help Desk Matrity Level Matrity Level N Mean Std. Deviation Initial Repeatable Standardized Managed Optimized Total Figre Strategic Plan Stats, by Help Desk Matrity Level Percentage of Instittions in Category 70% 60% 50% 40% 30% 20% % % Initial (N = 34) Repeatable (N = 126) Standardized (N = 180) Managed (N = 79) Optimized (N = 30) No plan Plan being developed Stand-alone plan Integrated plan Matrity Level Help Desk Matrity and ITSM Practices At their highest levels, both ITSM and CMMI have as their goals the alignment of IT services with bsiness needs. As one might expect, then, we fond a significant positive association between the help desk matrity levels or respondents reported and the nmber of basic IT service management practices they had implemented. As Table 10-4 illstrates, 1.40 was the mean nmber of or for basic ITSM practices implemented by respondents who evalated their help desk matrity as initial. That nmber increased steadily as help desk matrity increased ntil, at the optimized level, a mean of 3.33 of or for ITSM practices had been implemented. Matrity and the Help Desk/ Central IT Partnership As discssed in Chapter 9, it is important for the central IT organization to inclde the help desk in activities that may impact help desk client services. We asked respondents if their help desks personnel were adeqately inclded in central IT activities related to or for basic ITSM practices. The nmber of activities in which or respondents said the help desk was inclded was positively associated with help desk matrity. 108

11 Table Nmber of For Basic ITSM Practices Adopted, by Help Desk Matrity Level Matrity Level N Mean Std. Deviation Initial Repeatable Standardized Managed Optimized Total The mean nmber of activities reported by respondents whose help desks are at the initial level of matrity was Respondents at the other end, whose help desks are at the optimized level, reported adeqate engagement in twice that nmber, 3.13 activities ot of 4. This finding sggests that instittions with more matre help desks emphasize commnication and inclsiveness. Help Desk Matrity and Metrics Help desk matrity was positively associated with many of the findings related to help desk metrics that we reported earlier in this chapter. Respondents with optimized help desks reported the reglar analysis of three times more help desk metrics, comparing means, than those at the initial level of help desk matrity (see Table 10-5). Moreover, they said they reported those metrics to a mean of almost twice as many constitencies. Help Desk Matrity and Assessment of User Satisfaction As reported earlier in this chapter, respondents help desks varied widely in the nmber of methods they sed to assess ser satisfaction. Table 10-6 illstrates the positive association between that finding and the matrity of or respondents help desks. Help desks at initial matrity sed an average of 3.17 of the nine methods of assessment we asked abot. The nmber of methods rises with matrity level, leaping sbstantially from 4.21 at the managed level to 5.00 at optimized. This sggests that a focs on assessment goes hand in hand with help desk matrity. Help Desk Matrity and the Commnication of Costs and Vale Jst as commnication between central IT and the help desk was associated with help desk matrity, so was the help desk s commnication with its camps constitents abot help desk costs and vale. Table 10-7 docments this finding. As matrity level increases from initial to optimized, the mean levels of agreement abot all for aspects of the commnication of costs and vale rise. Means within a matrity level are fairly niform except that the mean for agreement that costs are well nderstood is always considerably lower. Smmary and Implications In general, or respondents help desks analyzed most freqently those metrics related to demand and problem resoltion time; they analyzed telephone-related metrics less freqently. Central IT staff and management were the primary constitencies to which help desk metrics were reglarly reported; or respondents said they reported metrics to top-level camps exectives less than half as often. These findings EDUCAUSE Center for Applied Research 109

12 Table Nmber of Metrics Analyzed Reglarly, by Help Desk Matrity Level Matrity Level N Mean Std. Deviation Initial Repeatable Standardized Managed Optimized Total Table Nmber of Methods for Assessing User Satisfaction, by Help Desk Matrity Level Matrity Level N Mean Std. Deviation Initial Repeatable Standardized Managed Optimized Total sggest that at the majority of respondent instittions, the help desk is internally focsed in its reporting relationships. Agreement that the help desk ses metrics effectively to improve ser service was fairly strong, with abot 40 percent agreeing, 30 percent disagreeing, and 30 percent netral. Stronger agreement was associated with greater nmbers of constitencies to which metrics were reported and greater nmbers of vehicles by which reporting is done, sggesting the interrelation of more se and more effective se of metrics. Also positively associated with agreement that the help desk ses metrics effectively were SLA se, how many docmented goals the help desk had, and the nmber of basic ITSM planning and management practices for which the central IT organization had adopted formal gidelines. Ths, the se of metrics appears to be part of a clster of deliberate and organized practices that help desks and their parent organizations employ together. Respondents more commonly sed informal methods of assessing ser satisfaction than formal ones, althogh nearly twothirds also sed formal srveys. A majority also sed Web feedback forms. Agreement was generally good that the help desk s vale is well docmented and well nderstood. Agreement that help desk costs were well docmented was variable, bt a majority of respondents disagreed that costs were well nderstood. As might be expected, the more strongly respondents agreed that camps expectations of the help desk were aligned with its resorces, the greater their agreement, in general, that help desk costs and vale were well nderstood. Respondents ratings of their help desks process matrity followed an approximately normal, bell-shaped distribtion, skewed a little to the immatre side bt with 40 percent saying their help desks were at the standardized level, in the middle of the range. Matrity was positively associated with several other factors, inclding more robst help desk toolsets for help desk admin- 110

13 Table Agreement Abot Commnication of Costs and Vale, by Help Desk Matrity Level Mean Level of Statement N Agreement* Std. Deviation Costs Are Well Docmented Initial Repeatable Standardized Managed Optimized Total Costs Are Well Understood Initial Repeatable Standardized Managed Optimized Total Vale Is Well Docmented Initial Repeatable Standardized Managed Optimized Total Vale Is Well Understood Initial Repeatable Standardized Managed Optimized Total *Scale: 1 = strongly disagree, 2 = disagree, 3 = netral, 4 = agree, 5 = strongly agree istrators, staff, and clients; the nmber of goals the help desk had adopted; the stats of strategic planning for the help desk; and alignment of camps expectations of the help desk with its resorces. It was also positively associated with the nmber of or for basic ITSM planning and management practices for which the IT organization had formal gidelines in place and the perceived adeqacy of the help desk staff s involvement in central IT s ITSM-related activities. Other positive associations with help desk matrity inclded the nmber of help desk metrics reglarly analyzed and the nmber of constitencies to which those metrics were reported. The mean nmber of methods sed to assess ser satisfac- EDUCAUSE Center for Applied Research 111

14 tion rose consistently with higher levels of help desk matrity, as did agreement that costs and vale of the help desk are well docmented and well nderstood. Endnote 1. CMMI Prodct Team, Capability Matrity Model Integration (CMMI), Version 1.1 (Pittsbrgh: Carnegie Mellon Software Engineering Institte, 2002),