Performance Management and Evaluation at Alberta Innovates Health Solutions: Defining the Research-to-Impact Framework

Performance Management and Evaluation at Alberta Innovates Health Solutions: Defining the Research-to-Impact Framework

Introduction Alberta Innovates Health Solutions (AIHS) is a publicly funded, board-governed, corporation that is part of Alberta s research and innovation system. Our mandate is to improve the health and wellbeing of Albertans through health research and innovation. On behalf of the Government of Alberta, we make substantial investments in health research and innovation. Part of our support is to assess the value of our investment programs and operations in contributing to health and socio-economic benefits. We do this through routine performance management and evaluation (PME) activities. This brief document intends to provide an overview of the AIHS health research-to-impact framework. AIHS is a learning organization and uses evaluation to learn what works so that it can continuously improve. Through our performance management and evaluation activities and the research-to-impact framework, we track progress and evaluate across the research-to-impact continuum. 1 Defining the Research-to-Impact Framework

The Building Blocks: Performance Management and Evaluation PME uses practice- and evidence-based approaches. Figure 1 indicates how the organization s strategic plans are translated into implementable actions and measures. The indicators and measures help inform what is and is not working and point to areas for process and performance improvement. The research-to-impact framework is based on the Buxton-Hanney Payback model as updated by the Canadian Academy of Health Science (CAHS) in a January 2009 report entitled: Making an Impact: A Preferred Framework and Indicators to Measure Returns on Investment in Health Research. It integrates the CAHS research logic model approach (inputs, activities, outputs, outcomes) with perspectives from the balanced scorecard model (Kaplan and Norton, 1996), i.e., the categorization of strategic organizational objectives into stakeholder, internal processes, enablers, and financial perspectives. Performance management, evaluation processes, and activities (e.g., cascading logic models from the organizational to program level) help inform the selection of appropriate progress markers and measures across the organization s research activities and the pathways to impact. Figure 1. AIHS performance management and evaluation. Strategic Planning 1. STAKEHOLDER PRIORITIES 2. MANDATE 3. VISION 4. STRATEGY 5. GOALS Implementation 6. OBJECTIVES 7. TOOLS SCORECARD & LOGIC MODELS 8. PROCESS & OUTCOME MEASUREMENT 9. BUSINESS INFORMATION MANAGEMENT SYSTEM 10. BUSINESS PLANNING Defining the Research-to-Impact Framework 2

From the Payback model to CAHS and to the AIHS Health Research-to-Impact Framework CAHS concluded that the application of the Buxton-Hanney Payback model (1996) was the best approach for capturing returns on investments in health research. The CAHS model of health research impact is built upon the strengths of the Payback model while attempting to address some of its limitations. As in the Payback model, the CAHS model utilizes a logic model framework that categorizes health research impacts into specific domains: 1) advancing knowledge, 2) building capacity, 3) informing decision making, 4) health impacts, and 5) broad socio-economic impacts. Each category has been summarized below: Advancing Knowledge: includes measures of research quality, activity, outreach, and structure. Research Capacity Building: includes measures pertaining to research personnel, leveraged (i.e., attracted) funding, and infrastructure. Informing Decision Making: represents one of the pathways from research to outcomes in health, wealth, and well being. Informing decision-making indicators fall into the sub-categories of: health related decision making (where health is broadly defined to include health care, public health, social care, and other health related decisions in areas such as environmental health); researchrelated decision making (how future health research is directed); healthproducts industry decision making; and public decision making. Health Impact: pertains to health status (e.g., quality of life as an important component of improved health), health system changes, and determinants of health. Determinants of health are further classified into three major subcategories: modifiable risk factors, environmental determinants, and modifiable social determinants. Broad Economic and Social Impacts: are classified into activity, commercialization, health benefit (e.g., specific costs of implementing research findings in the broad health system), wellbeing, and social benefit indicators (i.e., socio economic benefits). The CAHS model also describes the pathways to impact. This is represented in Figure 2 and demonstrates how research activity informs decision making, eventually resulting in changes in health, and economic and social prosperity (left to right arrow). The framework also shows how research has an impact on feedback upstream, potentially influencing the diffusion and impact of other research, and creating inputs for future research activity (right to left arrow) (CAHS 2009). 3 Defining the Research-to-Impact Framework

CANADIAN ACADEMY OF HEALTH SCIENCES EVALUATION FRAMEWORK Figure 2. Canadian Academy of Health Sciences Evaluation Framework Initiation and Diffusion of Health Research Impacts Health Status, Function, Well-being, Economic Conditions Global Research Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-Pillar Research Research Results Knowledge Pool Interactions/Feedback Health Industry Other Industries Government Research Agenda Public Information, Groups Health care Appropriateness, Access, etc. Prevention and Treatment Determinants of Health Improvements in Health and Well-being Economic and Social Prosperity Research Capacity Impacts Feed Back into Inputs for Future Research Payback Framework Topic Identification Selection Inputs Process Primary Outputs Dissemination Secondary Outputs Adoption Final Outcomes Defining the Research-to-Impact Framework 4

A Guide to AIHS s Health Research-to-Impact Framework A high level graphical representation of the health research-to-impact framework used by AIHS is provided in Figure 3. The framework tracks AIHS s external research funding and organizational level (program delivery or services) activities and results across the health research-to-impact continuum. The graphical representation of the AIHS health research-to-impact framework is inspired by the proposed CAHS model, but is contextualized within the Alberta Research and Innovation system, and specifically aligned to the Alberta Health Research Innovation Strategy (AHRIS) (AAET 2010) through which AIHS contributes to system level outcomes. The main driver of the framework is the overarching AIHS mission and vision. The framework integrates AIHS s strategic implementation process, which is characterized by broad stakeholder engagement approaches (e.g., cross sector community consultations), and evidence, all of which lead to the development and implementation of AIHS funding opportunities and programs and value-add service activities. In collaboration with partners and other key stakeholders, AIHS supports strategically focused research activity, manages its research investments, and delivers value-added services. The result is the development of a pool of highly skilled researchers and trainees, targeted infrastructure in priority areas, and improved receptor capacity within decision-making environments. Research results and knowledge are mobilized through knowledge exchange activities (e.g., interactions, collaborations, partnerships, networks, knowledge brokering, etc.) with relevant stakeholders (e.g., industry, government, research community etc.) across related sectors, provincially, nationally, and internationally. According to the model, the push of knowledge and the pull from end-users better informs decision making and future research agendas. The end products of health research results in more appropriate interventions in health care and in the health system. Ultimately, the benefits are improvements in health and well-being and greater socio-economic prosperity. Research and innovation impacts are realized over time. They may be described as short term, such as research results published in high quality journals; intermediate term, such as the use of new knowledge that results in cost-savings in a health service delivery process; or longer term, such as career paths for trainees and the development of new products and technologies. Outputs are attributed with more certainty the closer they are to the inputs provided by AIHS and become less easily attributable as we move towards the longer term outcomes, where the influence of AIHS is but one of the factors that result in such outcomes. 5 Defining the Research-to-Impact Framework

The linear appearance of the AIHS health research-to-impact framework does not entirely capture the simultaneous, interdependent, and dynamic feedback loops inherent in research and innovation systems. However, it attempts to depict how the results of supported health research investments feedback to inform future research activity (e.g., planning, investment strategies, policy, etc.) and how, through linkage and exchange mechanisms, research and innovation are diffused across the pathways from health research to impact. Performance management and evaluation processes and activities (e.g., cascading logic models from the organizational to program level) target the appropriate progress markers and measures along the pathway to impact. The necessary tools (e.g., case studies, bibliometrics, surveys, researcher reports, etc.) are used for monitoring and evaluating the extent to which AIHS s strategies and investments are contributing to its mission. AIHS VALUE FRAMEWORK Figure 3. AIHS Research to Impact Framework*. AIHS Mission & Vision Time: Short, Intermediate & Long Term Results (in Years) AIHS INPUT Stakeholder engagement & evidence to inform planning & investment strategies Support Focused Health Research Activity Deliver Value - adding Services Manage Research Investments Increased Knowledge Pool BETTER INFORMED DECISION MAKING Practices Policies Products Services Behaviours IMPROVEMENTS IN HEALTH Effective & Innovative Health Delivery System Enhanced Determinants of Health SOCIO-ECONOMIC PROSPERITY Input to Current & Future Research Capacity Building Advancing Knowledge Informing Decision Making Health Impacts Socioeconomic Impacts Diffusion of Innovation *Adapted from the Canadian Academy of Health Sciences (2009) model. Defining the Research-to-Impact Framework 6

Next steps for AIHS AIHS has applied the framework to existing programs and will continue to integrate the framework into its new research and innovation funding opportunities. Further investigation of network and knowledge translation theories, as well as testing of the pathway to impact theory of change, will continue to be explored in more detail. We ll continue to assess whether AIHS s programs and activities are achieving their objectives and intended outcomes, and whether the provision of evidence informs investment and decision making for improvement purposes. The great opportunity ahead for AIHS will be to work with our stakeholders to identify ways to best optimize impacts across the pathway and devise appropriate methods and tools for moving this type of research evaluation forward in the province s research and innovation system. References AAET, Alberta Advanced Education and Technology and Alberta Health and Wellness. (2010) Alberta s Health Research and Innovation Strategy, <http://eae. alberta.ca/media/277640/ahris_report_aug2010_web.pdf> accessed 1 June 2012. Buxton, M. and Hanney, S. (1996) How Can Payback from Health Services Research be Assessed? Journal of Health Services Research & Policy, 1/1: 35-43. CAHS, Canadian Academy of Health Sciences. (2009) Making an Impact: a Preferred Framework and Indicators to Measure Returns on Investment in Health Research. Ottawa, ON: CAHS. Kaplan, R. and Norton, D. (1996) The Balanced Scorecard: Translating Strategy into Action. Boston, MA: Harvard Business School Press. 7 Defining the Research-to-Impact Framework Funded by the Government of Alberta