Key Findings... 1 Section 1: Key Performance Indicators... 3 Institutional KPIs... 3 KPI Process... 4 KPI Categories... 4 Reporting KPIs...

Transcription

1 Abstract Business intelligence software is ubiquitous in private industry to measure, monitor, and communicate progress toward strategic goals; this process is being adopted in higher education. This report explores the degree to which institutional research offices are involved in institutions definitions and use of key performance indicators and the selection and use of business intelligence software.

2 Table of Contents Key Findings... 1 Section 1: Key Performance Indicators... 3 Institutional KPIs... 3 KPI Process... 4 KPI Categories... 4 Reporting KPIs... 5 Section 2: Business Intelligence Software... 6 Choosing Reporting and Analytics Software Tools... 6 Evaluating Software Tools for IR Work... 7 Section 3: Changing IR Role... 9 Section 4: IR Budget Appendix Survey Methodology and Respondent Characteristics Testing of Differences in Means Survey Instrument... 16

3 Introduction As higher education institutions focus on performance improvement, business intelligence garners increased attention, and the professionals who manage it are more visible. We now know that vast stores of data amount to a cacophony without the tools and expertise needed to strip away the noise and find real meaning that informs decisions. So how are institutions finding that meaning? What investments are they making in business intelligence and performance management? Who are the people most engaged in the processes of turning data into information? The Association for Institutional Research (AIR), with assistance from Ellucian, developed a survey to answer these questions. Ellucian s expertise in working with postsecondary education institutions provided useful insights that were incorporated into the survey of AIR members. Thanks go to the 542 respondents who gave us a grassroots view of these important issues. We were delighted to hear from a wide range of institutions including public four- year, private four- year, and public two- year colleges and universities. We now have a better sense of the tools, methodologies, and budgets committed to measuring and reporting performance. Thanks also go to Drs. Darlena Jones, Amelia Parnell, and Leah Ewing Ross of AIR who served as the lead researchers for this project. They developed the survey, compiled the data, conducted the analyses, and wrote the report. Mr. Christopher Coogan served as AIR s liaison with Ellucian in order to achieve this robust examination of a topic of great importance to higher education. We are pleased to share our findings and look forward to continued discussions on this topic. Key findings are reported below and are followed by the full research report. I hope these findings and the detailed information contained in this report encourage productive campus discussions that focus on improving the use of data to prove and improve higher education. Randy L. Swing, Ph.D. Executive Director Association for Institutional Research July 30, 2014 Tallahassee, Florida Key Findings Demand for information is exploding in higher education, and institutional researchers are the center of their institutions data and information hubs. Because of this need for data, business intelligence (the skills, applications, and technologies used to leverage an institution s data to support data- informed decision- making) is growing in popularity among senior leaders. The Association for Institutional Research (AIR) recently conducted a survey to determine institutions use of business intelligence (BI) and members roles in BI. Using KPIs: Most higher education institutions are adopting key performance indicators (KPIs), and institutional researchers are leaders in this arena. Survey evidence: PAGE 1

4 90% of respondents said KPIs are important to their institutions. 79% of respondents said their IR offices will be very involved in data collection to support KPI reporting within the next three years. 80% of respondents said their institutions track enrollment KPIs. 71% of respondents said their IR offices are the primary authorities on campus to collect KPI data. 41% of respondents said KPIs originated in the IR office. Selecting Analytical Software: In selecting institutional technology solutions, IR offices are part of decision making teams. Survey evidence: 42% of respondents said IR offices either participate in decisions or are the decision makers in choosing their institutions enterprise resource planning (ERP) systems. 68% of respondents said IR offices either participate in decisions or are the decision makers in choosing their institutions BI solutions. Changing IR Role: The IR role will change for some IR professionals in the near future. Survey evidence: 53% of respondents said they anticipate a role change in IR within the next three years. 38% of respondents predicted their roles will evolve toward being more strategic within the next three years. 27% of respondents noted that changes in analytical technologies will provide opportunities to conduct more advanced analytics. Budget: IR professionals are somewhat optimistic for stable or expanded office budgets in the near future. Survey evidence: 32% of respondents expect expansion of their IR office budgets over the next three years, 46% anticipate no changes in their office budgets, and 22% expect their budgets to be reduced. PAGE 2

5 Section 1: Key Performance Indicators Key Performance Indicators (KPIs) are metrics that provide leaders information to evaluate an institution s success or its progress toward a strategic goal. Examples of KPIs for higher education are enrollment, retention, and graduation rates. Institutional KPIs Nearly every respondent (90%) reported that KPIs are somewhat to very important to their institutions (Chart 1). When asked if KPIs are a major responsibility for IR offices, 64% said yes. Over half of respondents (57%) said it is very important that KPIs have consistent definitions across institutions to support benchmarking efforts. KPIs are a major responsibility for IR office It is very important for KPIs to have consistent definiaons for benchmarking Chart 1: Importance of KPIs KPIs are somewhat or very important to insatuaon Importance of KPIs 64% 57% 90% We asked respondents to identify the areas from which their KPIs originate. More than half of respondents reported that KPIs originate from the president/cabinet, 47% reported academic leadership, and 41% reported IR offices. External groups, such as state or federal agencies, play smaller roles (Chart 2). NOTE: This question was multi- response; responses total more than 100%. President/Cabinet Academic leadership Insatuaonal Research State Internal commibees Federal Department commibees Other funding organizaaons Other Chart 2: Originaaon of KPIs 9% 8% 7% 16% 51% 47% 41% 37% 32% 29% PAGE 3

6 KPI Process The KPI process includes identification of performance metrics, collection of data to measure current performance, reporting of data, strategic planning based on the outcomes of those metrics, and other institutional improvement initiatives. AIR asked survey respondents the degree to which their IR offices are involved in the KPI process. The percentages of respondents who noted that they are very involved in different stages of the KPI process are presented in Table 1. We also asked respondents what they anticipate their IR offices participation with KPIs will be in three years; they predict higher levels of future involvement for every stage of the KPI process. Creation of KPIs and work on other institutional performance improvement initiatives are the areas in which survey respondents expect the greatest increases in levels of involvement. Table 1: How involved was the IR office with the following processes at your institution? How involved do you expect to be three years from now? Current: Very involved 3 years from now: Very involved Creation of Key Performance Indicators (KPIs) 53% 69% 16% Data collection to support KPI reporting 73% 79% 6% Reporting on KPIs 69% 78% 9% Strategic planning institution- wide 53% 65% 12% Other institutional performance improvement initiatives 58% 75% 17% Change KPI Categories Performance metrics can be used in many areas of an institution. Respondents were asked to indicate in which areas their institutions track KPIs (Table 2). We found that most institutions track enrollment, while few track workforce management. We also asked respondents if their IR offices would have increased involvement over the next three years in these KPI categories. Table 2: Which of the following KPI categories does your institution currently track? How will the IR office s involvement change over the next three years? Institution Tracks KPI Increased involvement Enrollment 80% 57% Academics 73% 66% Recruitment/Admissions 65% 58% Finance and Administration 53% 39% Student Services/Student Life 35% 48% Alumni Relations/Advancement 35% 34% Workforce Management 22% 35% PAGE 4

7 Reporting KPIs Overwhelmingly, respondents said that their IR offices are the primary authorities to collect data on their institutions KPIs (Chart 3). Chart 3: Primary Authority to Collect KPI Data Insatuaonal research 71% Other Department leaders Academic/Provost Office Informaaon Technology Operaaons/COO Finance/CFO 10% 6% 6% 4% 2% 1% 1% Chart 4: Primary Reporters of KPIs Insatuaonal Research Academic/Provost Office 9% President 8% Department Leaders 7% Finance/CFO 6% Other 5% Operaaons/COO 3% VP Student Services 3% Informaaon Technology 2% 58% When asked who reports information on KPIs at their institutions, 58% of respondents cited their IR offices (Chart 4.) Other KPI reporters include provosts, presidents, and department leaders. We asked respondents to identify all of the areas to which KPIs are reported (Chart 5.) Two- thirds of respondents said their institutions KPIs are reported to leadership and 40% said their accrediting bodies received KPI data. Nearly one- third (29%) of respondents said KPI data are reported on their websites and to the public. NOTE: This question was multi- response; responses total more than 100%. Chart 5: Reporang KPIs Leadership Accrediang bodies 40% State/federal agencies 39% Faculty 34% Staff focused on measurement 31% Website 29% The public 29% Funding agencies 16% Other 8% 67% PAGE 5

8 Section 2: Business Intelligence Software Choosing Reporting and Analytics Software Tools We asked respondents if their IR offices are part of decision- making processes to select their institutions enterprise resource planning (ERP) and business intelligence (BI) reporting and analytics software tools (Chart 6). We found that 42% of respondents IR offices either participate in decisions or are the decision makers in selecting their institutions ERP systems. We also found that 68% of respondents said their IR offices either participate in decisions or are the decision makers in selecting their institutions BI solutions. Chart 6: Choosing ERP and BI Solware 1% 18% 41% 50% 58% 32% ERP system BI soluaon Decision maker 1% 18% Paracipated 41% 50% No involvement 58% 32% Regarding institutions reporting and analytics tools/software, 76% of respondents IR offices either participate in decisions or are primary decision makers (Chart 7). When respondents were asked what they anticipate in terms of their IR offices influence on future purchases of reporting and analytics software tools, 96% believe their IR offices will participate in the decisions or will be decision makers. Chart 7: Choosing Reporang and Analyacs Solware Tools 26% 21% 50% 24% Current insatuaon tools 75% 4% Future IR tools Decision maker 26% 21% Paracipated 50% 75% No involvement 24% 4% PAGE 6

9 Evaluating Software Tools for IR Work We asked respondents to identify the ideal features for software tools (Chart 8.) Nearly three- fourths noted that they desire a tool that provides access to all required data, and 69% would like the ability to capture historical data and decisions. NOTE: This question was multi- response; responses total more than 100%. Chart 8: Features of an Ideal Solware Tool Access to all required data 74% Ability to capture historical data and decisions 69% Ability to engage all major departments electronically 53% Out of the box integraaon with ERP 50% Built- in KPIs and metrics for tracking performance 48% Respondents described the features their ideal software tools would contain. In their words Be user friendly to someone who doesn't know how to code or do SQL. I should be able to build queries and reports as simply as it is to use Office or surf the web. Publish directly to and provide interactive use on websites without specialized web programming. Ease of use and ease of learning. Organization and easy search for reports etc. We don't have a high- powered predictive analytics tool, and that's what we need. Be able to integrate data from multiple sources and be interactive in filtering fields (users get to pick fields and criteria) and also be able to do a drill down (e.g. start with university overall enrollment and then add layers like UG/Grad, In state- Out- of- state, Male/Female, etc.) and be available via web interfaces It should readily interface with other databases. It should allow for real- time (or at least near- real time) visualizations of data out of the box. It should provide a user interface designed to be easy for a beginner to learn, with options available for more advanced users. PAGE 7

10 We presented survey respondents with a list of KPI tools currently on the market and asked them to indicate their satisfaction with the tools they use (Table 3). Table 3: Software Tools % Respondents whose institutions use tool for KPIs Satisfaction of respondent using tool in IR work Dissatisfied (Answer = 1, 2) Neutral (Answer = 3) Satisfied (Answer = 4, 5) Microsoft Excel 60% 2% 7% 90% Business Objects (SAP) 6% 10% 25% 65% Entrinsik 5% 25% 13% 63% IBM Cognos 14% 25% 14% 61% Crystal Reports 16% 7% 36% 57% SharePoint 16% 12% 35% 53% Blackboard Analytics 8% 15% 35% 50% Nuventive TracDat 2% 0% 50% 50% Evisions Argos 11% 23% 46% 31% We asked respondents to list other software tools used in their IR work, which resulted in a list of 34 additional tools; SPSS, SAS, Access, and Tableau were the most common. We asked respondents to identify all barriers they experience when using their current software tools for IR work (Chart 9). The need for more training and the cost of the tools were noted as the most common barriers. Chart 9: Barriers to Using Tools Need more training Cost 39% 38% The right data aren't available Inability to engage with all insatuaonal departments Solware is not user- friendly Lack of historical data and decision capture 33% 30% 29% 28% No enterprise resource planning integraaon 24% PAGE 8

11 Section 3: Changing IR Role We asked respondents what changes they expect in their IR roles in the next three years. The predominant theme in respondents predictions was that IR will have a more strategic role (Table 4). Table 4: Changing IR Role in Next Three Years More strategic/planning More data/analytics focused More assessment focused Use of more BI tools More reporting More technical More involvement in educating users More of a support role Leaving the job In a new position Don't know No change % of Responses 38% 17% 8% 7% 6% 1% 1% 1% 1% 1% 5% 5% In their words The role of IR within the university setting is changing particularly in the aspect of strategic planning, data support for decision making. Involved in more strategic planning and assessment activities More automated required reporting to different agencies. More time spent answering and asking questions related to student success and college goals. My role will evolve into more of a support capacity. We have developed automated program review report application and we have implemented a more user- friendly Academic Assessment tool. I believe that we will be more in the area of outcomes assessment. IR will take over a more direct role in the long- term planning of BI with also hands- on development of BI (ODS/EDW build- out, meta- data, report /dashboard development, data governance). IR will go from strictly BI consumer to a key player in expanding and maintaining BI and NEIU. PAGE 9

12 We asked respondents to reflect on how advances in analytical technologies will change the responsibilities of their jobs. The predominant theme in respondents predictions was that IR professionals will have the abilities to conduct more analyses (Table 5). Table 5. Change in Responsibilities Due to Advances in Analytical Technologies Allow for more analyses Improve efficiencies More reporting Don't know/no answer General statement Other users accessing data Need to adapt to new technologies No changes to job New hires with different skills Automated reporting Data story teller Managing data More programming design Needs training Pushing beyond current tasks Support role % of Responses 27% 10% 10% 8% 8% 8% 6% 6% 4% 2% 2% 2% 2% 2% 2% 2% In their words I will do less simple ad hoc and more in- depth analysis of data. I'm looking forward to the upcoming changes they can't come soon enough. The hope is that analytics allows our office to spend less time on standard and ad hoc enrollment reporting (because it is much less manual and more seamless) and allows our office to spend more time on analysis, evaluation, and assessment. It will make data readily available to end users. It will ascertain the accuracy of data so that we can always have consistent data report no matter the time of the year the data is pulled from the student administration system. If done properly, it will make us more productive (better and faster analysis without the issues of data gathering and cleansing, which now take up more than half of the time). New hires must be more proficient and experienced in using such technologies. Will need to walk in the door with the basic training already completed. PAGE 10

13 Section 4: IR Budget We asked respondents to anticipate how their IR offices budgets will change over the next three years; the responses were very different by sector (Chart 10). All Respondents: We found 46% of respondents expect no change in their budgets and 32% expect their budget to expand over the next three years. Public 4- Year Institutions: Disaggregating by U.S. sector, we find more than half of respondents from public 4- year institutions believe their budgets will stay the same and 28% anticipate expansion of their budgets over the next three years. Public 2- Year Institutions: Over 90% of respondents from public 2- year institutions in the U.S. anticipate their budgets will stay the same or expand over the next three years. Private 4- Year, Not- for- Profit Institutions: Respondents from private, 4- year, not- for- profit institutions in the U.S. were less positive. We found 50% of respondents expect their budgets to shrink over the next three years. NOTE: We only report responses from the three sectors listed above because of lack of responses in other sectors. Respondents from public 4- year institutions rated this item statistically higher than respondents from private, 4- year, not- for- profit institutions (p <.05). Chart 10: Anacipated Change to IR Budget Over Next Three Years 3% 2% 6% 10% 29% 26% 43% 40% 46% 52% 43% 35% 17% 6% 5% 4% 2% All Respondents Public, 4- Year Public, 2- Year Private 4- Year, Non- profit Expand by 25% or more 3% 2% 6% 0% Expand up to 25% 29% 26% 43% 10% Stay the same 46% 52% 43% 40% Reduce up to 25% 17% 16% 6% 35% Reduce by 25% or more 5% 4% 2% 15% 16% 15% PAGE 11

14 We asked respondents to approximate the amount spent annually by their IR offices on analytical software tools. The largest segment, 42%, noted that less than $1,000 is spent annually, and only 25% of respondents reported that their offices spend $5,000 or more annually (Chart 11). Chart 11: Approximate Annual Amount Spent by IR Office on Analyacal Solware In a separate, but related question, we asked respondents to identify the amount of the single largest purchase made by their IR offices in the past year. One- quarter noted that they spent less than $1,000 and 46% said $5,000 or more. Up to $1,000 $1,001 to $5,000 $5,001 to $9,999 $10,000 or more 11% 14% 32% 42% Chart 12: Analyacal Solware Expenditures from IR Office Budget No, 31% Yes, all, 15% We asked respondents whether analytical software expenditures come out of IR offices budgets. Nearly one- third (31%) said no and 15% reported that all expenditures come from IR office budgets (Chart 12). Yes, some, 54% PAGE 12

15 Closing There s no doubt about it: institutional research professionals have an increasingly vital role in helping institutions identify, understand, and act upon opportunities for performance improvement. More than half of survey respondents said they foresaw their roles changing in the future, and nearly 40% said they expected to adopt a more strategic role in their institutions decision- making processes. That all makes sense given the overwhelming emphasis on key performance indicators today. A full 90% of respondents said KPIs are important to their institution. That importance is here to stay. And that s exciting. With the right tools, training, and expertise, institutions can get the insights they need to truly elevate and increase overall performance to better serve students. Ellucian Perform is for campus leaders who have multiple sources of data and want to drive a culture of data- based decision making throughout the entire institution. To learn more about the industry s first- to- market platform for establishing and maintaining a data- driven culture of performance and how this solution can significantly support the changing role of IR, visit Ellucian.com. Wayne Bovier, Vice President of Product Management, Ellucian Association for Institutional Research 1435 E. Piedmont Drive, Suite 211, Tallahassee, FL (phone), (fax) Ellucian Headquarters: 4375 Fair Lakes Ct, Fairfax, VA 22033, USA Phone: To download copies of this research, visit PAGE 13

16 Appendix Survey Methodology and Respondent Characteristics Survey metrics were written by assessment experts at AIR in collaboration with staff from Ellucian, a company that specializes in business intelligence (BI) software. Sixty- five questions on the survey explored the types of key performance indicators (KPIs) used on respondents campuses, use of BI software, and respondents personal experiences with BI software. AIR randomly selected 1,546 members of the Association to participate in the study. The survey participants were divided into three groups, and the survey was divided into three sections; each survey group received one section of the survey in order to reduce respondents survey burden. An online surveying system was used to collect data April 22 May 6, One invitation and two reminder s were sent to survey participants. We collected 542 responses for a 35% response rate. Of the 542 responses, 536 individuals self- identified their IR roles and the locations of their institutions (Table 6). Location of Institution Table 6: IR Role by Institution Location U.S./U.S. territories Canada Outside the U.S. and Canada Total College or university, in an IR office College or university, not in an IR office System office Non- profit organization Retired Other TOTAL For the purpose of this report, we focused on the 424 respondents who work in college or university IR offices in the U.S./U.S. territories or Canada. (In the report, reference to institutions in the U.S. includes U.S. territories.) The distribution of the 424 responses by survey is shown in Table 7. Table 7: Survey # Responded Response Rate Survey % Survey % Survey % TOTAL 424 PAGE 14

17 Senior institutional research (IR)/institutional effectiveness (IE) officers comprised nearly 60% of the response pool (Table 8). Note that 13 respondents failed to answer this item. NOTE: Due to rounding errors, values may appear to not be equal to 100%. Table 8: Respondent's Role # of Respondents % of Respondents Senior IR/IE Officer % Associate/Assistant IR/IR Officer 43 11% IR Analyst/Researcher 98 24% Senior Assessment Officer 2 1% Assessment analyst/researcher 15 3% Other 12 3% Total 410 Respondents who work at post- secondary institutions in the U.S. were matched with their institutions IPEDS sectors. The largest sector represented in the data set was private, not- for- profit, 4- year institutions followed by public 4- year institutions (Table 9.) Table 9: Institutional Sector # of Respondents % of Respondents Private not- for- profit, 4- year or above % Public, 4- year or above % Public, 2- year 87 22% Private for- profit, 4- year or above 14 4% Private not- for- profit, 2- year 1 0% Total 401 Testing of Differences in Means We performed t- tests on survey metrics between respondents in different sectors. There were no statistical differences in means by sector except for the metric describing the anticipated change in IR budget over the next three years, as noted. In that metric, respondents from public 4- year institutions rated that item statistically higher than respondents from private, 4- year, not- for- profit institutions. PAGE 15

18 Survey Instrument Which best describes your current place of employment? Where is your primary work location? How involved should IR offices be in choosing their institutions business intelligence solutions? How involved should an IR office be in choosing their institutions reporting and analytics tools/software? How do you expect an institution s IR office budget to change in the next 3 years? Which best describes your role in Institutional Research? How involved was your IR office in choosing your institution s business intelligence solution? How involved was your IR office in choosing your institution s reporting and analytics tools/software? College or university, in an Institutional Research office College or university, not in an Institutional Research office System office Non- profit organization I am retired In the U.S./U.S. territories In Canada Outside of the U.S. and Canada Should have primary responsibility for decision Should participate in decision- making Should have no involvement Should have primary responsibility for decision Should participate in decision- making Should have no involvement Reduce by 25% or more Reduce up to 25% Stay the same Expand up to 25% Expand by 25% or more Senior IR/IE Officer Associate/Assistant IR/IR Officer IR Analyst/Researcher Senior Assessment Officer Assessment analyst/researcher Primary responsibility for decision Participated in decision- making No involvement Primary responsibility for decision Participated in decision- making No involvement PAGE 16

19 Primary responsibility for decision How involved was your IR office in Participated in decision- making choosing your institution s ERP No involvement (enterprise resource planning) system (e.g., Ellucian, Jenzabar, PeopleSoft)? Institution doesn't have an ERP system How do you see your role changing in the next few years? That is, do you anticipate that it will be more or less strategic, encompass different responsibilities, etc.? Information Technology How frequently do you interact with each of the following? (Scale = 1) Daily, 2) Weekly, 3) Monthly, 4) Less than monthly, 5) Rarely or never) How involved was the IR office with the following processes at your institution? (Scale = 1) Not involved at all, 2) Somewhat involved, 3) Very involved, 4) Institution does not use KPIs, 5) ) Finance Registrar Academic Department Leaders Administrative Department Leaders Senior Leaders (CEO, CAO, etc.) Cabinet/Board of Trustees Creation of Key Performance Indicators (KPIs) Data collection to support KPI reporting Reporting on KPIs Strategic planning institution- wide Other institutional performance improvement initiatives How involved will the IR office expect to be with these processes in the next 3 years? (Scale = 1) Not involved at all, 2) Somewhat involved, 3) Very involved, 4) Institution does not use KPIs, 5) ) Are KPIs (Key Performance Indicators) a major responsibility for the IR office? Where do the KPIs used at your institution originate? Please select all that apply. Creation of Key Performance Indicators (KPIs) Reporting on KPIs Data collection to support KPI reporting Strategic planning Other institutional performance improvement initiatives Yes No State Federal Other funding organizations President/Cabinet Academic leadership Institutional Research Internal committees Department committees PAGE 17

20 Who has the primary authority to collect data on KPIs? Who is the primary reporter on KPIs? Where are KPIs reported? Please select all that apply. How are KPIs being used at your institution? Institutional Research Information Technology Department Leaders Finance/CFO Operations/COO Academic/Provost Office Institutional Research President Information Technology VP Student Services Department Leaders Finance/CFO Operations/COO Academic/Provost Office Leadership Staff focused on measurement Faculty The public Accrediting bodies Funding agencies Government (state/federal) agencies Website Formal strategic planning Obtain funding resources Accreditation IPEDS reporting Other external reporting PAGE 18

21 What software tools does your institution currently use for tracking KPIs (Key Performance Indicators) and reporting on data? Please select all that apply. Blackboard Analytics Business Objects (SAP) Crystal Reports Entrinsik Evisions Argos IBM Cognos Microsoft Excel Nuventive TracDat SharePoint Don't know Please list any other software tools used for IR tasks with which you are familiar. Thinking of the ideal software tool for your role, what features would it include? Please select all that apply. Please indicate which of the following KPI categories your institution currently tracks. Please select all that apply. How will the IR office s involvement with the following KPI categories change in the next 3 years? (Scale = 1) Decreased involvement, 2) Involvement will stay the same, 3) Increase involvement, 4) Institution doesn't use this category, 5) Don't know) Access to all required data Out- of- the- box integration with ERP Ability to engage all major departments electronically Ability to capture historical data and decisions Built- in KPIs and metrics for tracking performance Recruitment/Admissions Enrollment Student Services/Student Life Academics Alumni Relations/Advancement Finance and Administration Workforce Management None of these Recruitment/Admissions Enrollment Academics Alumni Relations/Advancement Finance and Administration Workforce Management What is the level of importance placed on KPIs by your institution? (Scale = 1) Not at all important, 2) Somewhat important, 3) Very important, 4) My institution doesn't use KPIs, 5) Don't know) PAGE 19

22 How important is having KPIs defined across institutions to support benchmarking? That is, how useful would common definitions across institutions for KPIs be in order to support comparisons? (Scale = 1) Not at all important, 2) Somewhat important, 3) Very important, 4) My institution doesn't use KPIs, 5) Don't know) What will be your IR office s future involvement with these KPIs categories? Blackboard Analytics Thinking about the software tools that you use for Institutional Research, how satisfied are you with each? (Scale = 1) Very dissatisfied, 2) Dissatisfied, 3) Neutral, 4) Satisfied, 5) Very satisfied, 6) Don't use this) Business Objects (SAP) Crystal Reports Entrinsik Evisions Argos IBM Cognos Microsoft Excel Nuventive TracDat SharePoint What would your ideal software tool be able to do that your current tool doesn't? Please explain. Thinking about the software tools you are currently using for Institutional Research, what are the barriers to using these tools? Please select all that apply. Which of the following would help you to retrieve data from your systems? Please select all that apply. How do you see the budget for your IR office changing in the next 3 years? There are no barriers to the tools I currently use The right data aren't available Need more training Software is not user- friendly Cost No enterprise resource planning integration Lack of historical data and decision capture Inability to engage with all institutional departments Additional services engagement with system vendor Additional services from your Information Technology department Workarounds (e.g., export data to Excel) Manual data collection/entry Reduce by 25% or more Reduce up to 25% Stay the same Expand up to 25% Expand by 25% or more Don't know PAGE 20

23 What is the approximate value of largest purchase your IR office has authorized in the past 12 months? Approximately how much does your IR office spend annually on analytical software tools? Do analytical software expenditures come out of the IR office's budget? What kind of influence do you think the IR office will have in the selection of analytical software tools for Institutional Research in the next 3 years? Up to $1,000 $1,001 to $5,000 $5,001 to $9,999 $10,000 or more Don't know Less than $1,000 $1,000 to $4,999 $5,000 to $9,999 $10,000 or more Yes, all Yes, some No Will have primary responsibility for decision Will participate in decision- making Will have no involvement How do you think that the advances in analytical technologies will change the responsibilities of your job? PAGE 21