Metric Development Form Part 1. About You Please fill out the following table: Date: Division Unit: Prepared by: Part 2. Background The purpose of this form is to help each Division manager organize their thoughts about what metrics best tell the story of their unit, and how their unit supports the T&C mission. T&C Mission To provide and support technology and communication that enhances the educational experience and University operations. Root Question To Answer: The question we are trying to answer with these metrics is: How effective are we at enhancing the educational experience and University operations for our customers? Let s deconstruct this statement and look it in a bit more detail: 1. What is effective? For our purposes, it means how well are we able to accomplish a purpose or produce an expected result. Effectiveness depends on such factors as responsiveness, speed, availability, accuracy, predictability, and openness. It also includes looking at how the systems & services we offer are used, and how satisfied our customers are with them. 2. educational experience encompasses all activities related to teaching, learning, research and student life. First and foremost, think about the complete life cycle of a student in their interaction with the University, and how we support the University s interaction with them, from the moment they become interested in CI, to the moment they graduate or become an alum. 3. University operations encompasses activities which involve the day to day function of University business and organization. University operations may or may not directly support the educational experience, and most often relates to behind the scenes work done to increase the capacity and/or capability of its employees to support the educational experience, and to keep the University running smoothly in general. 4. Our customers are current and prospective students, faculty, staff & administrators, parents, alumni, affiliates, visitors and the general public. As part of staff and administrators, sometimes T&C is its own customer! 1 P age
Part 3. Visualization Exercise In the space below on this page, draw a picture or pictures (using photos, clipart, text/phrases, etc), which answers the question: what does it mean for my division unit to be effective? 2 P age
Part 4. Determining measures for your unit Based on the qualities you visualized in Part 1 of the exercise, in this section, you will define measures that help answer this question: How effective is my division unit at enhancing the educational experience and University operations for our customers? Please complete the following tables for at least 3 (but no more than 7) measures that would help you answer this question. Measure #1 Measure Description How does this measure help tell the story of your division unit? How does this measure help tell the story of the Division? How will the data be gathered? Who will gather & store the data? gathered? Where & how will the data be stored? analyzed & reported? Who will analyze & report the data? What is the expected minimum normal range What is the expected maximum normal range How should this measure should NOT be used? Describe any known (or desired) benchmarks that could be used to enable comparative analysis Write a 1 sentence description of the measure. See the Appendix for examples of measures. Write 1 2 sentences which describe how this measure will help your unit improve delivery speed, accuracy, availability, usage, or customer satisfaction. Write 1 2 sentences which describe how this measure will help the Division enhance the education experience or University operations. If a measure supports a specific item from the strategic plan, please list it here. Type your answer here. Examples: observation, automated tools, customer survey, focus groups, in person interviews Enter the name of person(s) responsible for obtaining and storing the data Data gathering interval must be less than 1 year. Specify location (e.g., network shared drive) and format (Excel, log files, etc) for data. Data analysis and reporting interval must be less than 1 year. Enter the name of person(s) responsible for analyzing and reporting the data. effectiveness for the measure (example: 15 users, 50% usage, 10 minutes, etc). Enter Unknown if no regular range is known/can be effectiveness for the measure (example: 200 users, 90% usage, 2 hours, etc). Enter Unknown if no regular range is known/can be If applicable, describe in brief narrative how the measure or corresponding data should not be used. Enter N/A if not applicable. List any benchmarks that can be used to help gauge progress. Example: CSU ATI annual campus evaluation for web accessibility; Educause Core Data Survey/Core Data Service. See Appendix C for more information. 3 P age
Measure #2 Measure Description How does this measure help tell the story of your division unit? How does this measure help tell the story of the Division? How will the data be gathered? Who will gather & store the data? gathered? Where & how will the data be stored? analyzed & reported? Who will analyze & report the data? What is the expected minimum normal range What is the expected maximum normal range How should this measure should NOT be used? Describe any known (or desired) benchmarks that could be used to enable comparative analysis Write a 1 sentence description of the measure. See the Appendix for examples of measures. Write 1 2 sentences which describe how this measure will help your unit improve delivery speed, accuracy, availability, usage, or customer satisfaction. Write 1 2 sentences which describe how this measure will help the Division enhance the education experience or University operations. If a measure supports a specific item from the strategic plan, please list it here. Type your answer here. Examples: observation, automated tools, customer survey, focus groups, in person interviews Enter the name of person(s) responsible for obtaining and storing the data Data gathering interval must be less than 1 year. Specify location (e.g., network shared drive) and format (Excel, log files, etc) for data. Data analysis and reporting interval must be less than 1 year. Enter the name of person(s) responsible for analyzing and reporting the data. effectiveness for the measure (example: 15 users, 50% usage, 10 minutes, etc). Enter Unknown if no regular range is known/can be effectiveness for the measure (example: 200 users, 90% usage, 2 hours, etc). Enter Unknown if no regular range is known/can be If applicable, describe in brief narrative how the measure or corresponding data should not be used. Enter N/A if not applicable. List any benchmarks that can be used to help gauge progress. Example: CSU ATI annual campus evaluation for web accessibility; Educause Core Data Survey/Core Data Service. See Appendix C for more information. 4 P age
Measure #3 Measure Description How does this measure help tell the story of your division unit? How does this measure help tell the story of the Division? How will the data be gathered? Who will gather & store the data? gathered? Where & how will the data be stored? analyzed & reported? Who will analyze & report the data? What is the expected minimum normal range What is the expected maximum normal range How should this measure should NOT be used? Describe any known (or desired) benchmarks that could be used to enable comparative analysis Write a 1 sentence description of the measure. See the Appendix for examples of measures. Write 1 2 sentences which describe how this measure will help your unit improve delivery speed, accuracy, availability, usage, or customer satisfaction. Write 1 2 sentences which describe how this measure will help the Division enhance the education experience or University operations. If a measure supports a specific item from the strategic plan, please list it here. Type your answer here. Examples: observation, automated tools, customer survey, focus groups, in person interviews Enter the name of person(s) responsible for obtaining and storing the data Data gathering interval must be less than 1 year. Specify location (e.g., network shared drive) and format (Excel, log files, etc) for data. Data analysis and reporting interval must be less than 1 year. Enter the name of person(s) responsible for analyzing and reporting the data. effectiveness for the measure (example: 15 users, 50% usage, 10 minutes, etc). Enter Unknown if no regular range is known/can be effectiveness for the measure (example: 200 users, 90% usage, 2 hours, etc). Enter Unknown if no regular range is known/can be If applicable, describe in brief narrative how the measure or corresponding data should not be used. Enter N/A if not applicable. List any benchmarks that can be used to help gauge progress. Example: CSU ATI annual campus evaluation for web accessibility; Educause Core Data Survey/Core Data Service. See Appendix C for more information. 5 P age
Appendix Appendix A. Examples of Measures These examples are organized by measure categories, and designed to help division managers get ideas for measures which respond to our root question. Delivery Measure Examples (How well are we delivering service to our customers? Are our services available when the customer wants them? How long does it take to deliver our services to our customers? Do we deliver what we say, or are there errors involved?) % of XYZ system/service downtime % of XYZ system/service uptime # of hours of XYZ system/service during maintenance blackout periods # of emergency change requests incidents key blackout periods % of urgent/emergency change requests # of campus customizations of PeopleSoft % of chartered projects completed on time % of chartered projects completed on budget % of successfully completed RFCs % of RFCs that have caused major issues % of repeat calls to the Help Desk for the same incident # of Help Desk calls missed % of tickets resolved in 24 hours or less % of tickets resolved in 1 hour or less % of tickets resolved in 5 days or less % of services provided through external (offcampus) IT % of backups restored in under 30 minutes % of press release requests which become press releases Usage Measures Examples (Are our services being used?) % of active projects for external (non T&C customers) % of Help Desk tickets resolved using knowledge base % of CI web site visits using mobile devices # of PeopleSoft security requests # of Help Desk tickets (from ABC unit/program, for customer XYZ, etc) # of logins to XYZ system/service # of myci users during registration periods # of T&C web site visits # of password reset assistance requests at the Help Desk Top 10 most visited web pages % of available network storage space in use # of backup requests # of courses in CI Learn # of active users in XYZ system # of project requests via project request form # of press releases requests # of workshop/training participants # of downloads of regular publications # of new web sites created # of brochures published # of hours of video footage captioned Customer Satisfaction Measures (Are our customers satisfied with the services and products we provide?) customer satisfaction question The staff in unit XYZ is knowledgeable customer satisfaction question XYZ system/service has the tools I need to do my job customer satisfaction question XYZ system/service is easy to use customer satisfaction question XYZ system/service is fast customer satisfaction question Projects I ve requested are completed by T&C staff on time customer satisfaction question I am satisfied with how the CI web site works on a mobile device customer satisfaction question I am happy with XYZ publication customer satisfaction question I am happy with the result of with DEF project More examples of measures can be found at: http://itbenchmark.files.wordpress.com/2010/06/sample_metrics_for_itil_processes.pdf Appendix B. Possible Areas to Analyze 6 P age
Planning, Policy & Assessment this plan, business practices, external requirements, strategic planning System Design Networking and Telecommunication Coding and Programming Project and Portfolio Management Security and Safety Resource Management hardware and software inventories Support (& customer confidence) Desktop PC Management Mobile device management Instructional Technology Classroom Technology IT Operations Human Resorce Management retention, satisfaction Training: inward and outward facing Procurement Budget Communication (technology specific) Communication & Marketing (unit) Appendix C. Benchmarks available from Educause Core Data Service Staffing counts Lists of services provided by IT Overall costs of IT Cost of each service provided by IT Distribution of IT spending (staffing, software, hardware, etc.) Funding sources of IT budget Staffing sourcing (part time, full time, students, contractors, etc.) Key performance indicators (KPIs) to measure IT service quality and efficiency KPIs based on existing industry standards Counts of central IT hardware (servers, network nodes, etc.) Counts of end user devices (laptops, tablets, etc.) Sourcing arrangements for IT services (e.g., central, distributed, outsourced, etc.) Names of vendors/applications used for IT services Measures of user satisfaction Data about decentralized IT services About this document Revised: 2012 01 23 Created on: 2012 01 23 by Peter Mosinskis 7 P age