Using Value of Information Analysis to Prioritise Health Research: Some Lessons from Recent UK Experience

Transcription

1 Using Value of Information Analysis to Prioritise Health Research: Some Lessons from Recent UK Experience Claxton K. a,b and Sculpher M.J. a Centre for Health Economics and Department of Economics and Related Studies, University of York. Acknowledgments: We would like to acknowledge all those involved in the two pilot studies which are discussed in this paper: Eggington S., Ginnelly L., Griffin S., McCabe C., Palmer S., Philips Z., Tappenden P. and Wailoo A. Abstract Decisions to adopt, reimburse or issue guidance on the use of health technologies are increasingly being informed by explicit cost-effectiveness analyses of the alternative interventions. Health care systems also invest heavily in research and development to support these decisions. However, the increasing transparency of adoption and reimbursement decisions, based on formal analysis, contrast sharply with the decisions about research priorities and commissioning. This, despite the fact that formal measures of the value of evidence generated by research are readily available. In this paper, the principles of value of information analysis are briefly outlined before discussing the results of two recent opportunities to apply these methods directly to inform policy decisions about research priorities in the UK. These include a pilot study for the UK National Co-ordinating Centre for Health Technology Assessment (NCCHTA) and a pilot study for the National Institute for Health and Clinical Excellence (NICE). The implications of the results of the pilots for research priorities and the type of research needed are explored. Some of the methodological and policy challenges are discussed.

2 1. Introduction Decisions to adopt, reimburse or issue guidance on the use of health technologies are increasingly being informed by explicit cost-effectiveness analyses of the alternative interventions. 1 This requires an analytic framework which can represent these decision problems explicitly, combine evidence from a range of sources and facilitate the extrapolation of costs and effects over time and between patient groups and clinical settings. 2,3 Such an analytic framework can inform both the decision to adopt or reject a technology and whether additional research is required to support this decision in the future. Health care systems invest heavily in research and development. For example the UK Department of Health currently allocates approximately 650 million per year (2005/06) through a portfolio of National Research Programmes and the NHS R&D Programme. 4 The US is estimated to devote $31 billion of federal funds to research and development in These resources are substantial and could be made available to support provision of health care. However, the increasing transparency of adoption and reimbursement decisions, based on formal analysis, contrast sharply with the decisions about research priorities and commissioning. This, despite the fact that formal measures of the value of evidence generated by research are readily available, and evidence that the value of additional information can be substantial and in some circumstances offer greater benefits than the decision to adopt a technology itself. The decision to adopt a technology should not be separated from the question of whether the evidence to support such a decision is sufficient. However, those institutions with the remit for making adoption and reimbursement decisions are often separated from those responsible for prioritising and commissioning research. The former often come under closer public scrutiny than the latter and have, therefore, been more willing to adopt transparent and explicit approaches to their decisions. 3 A number of methods for setting priorities in research and development of health care technologies have been proposed, and some have been used to identify priority areas for research. These include measures of the burden of disease, or the technology 6, 7 measures of the expected payback from research, 8-10 and estimates of the welfare losses due to variations in clinical practice. 11 However, each of these proposed 2

3 methods has serious methodological problems. Firstly, all of these approaches view research simply as a means of changing clinical practice. But there are many ways to change clinical practice and conducting research may not be the most effective or cost effective means to do so. The real value of research is to generate information about what clinical practice should be. The question of how to implement research findings is a separate, although related, issue. Indeed, measures of payback or welfare losses due to variations in clinical practice require the analysis to identify the appropriate utilisation of a technology a priori. Therefore, these methods implicitly assume that there is no uncertainty surrounding the decision that the proposed research is supposed to inform. Secondly, these approaches, particularly measures of burden, attempt to identify research priorities using aggregate measures across broad clinical areas. However, the information generated by evaluative research is only valuable if it informs specific clinical decisions for specific groups of patients. The value of research in a clinical area is simply made up of the value of research about each of the constituent clinical decision problems faced within that area. Therefore, simply because aggregate measures such as burden of disease may suggest a clinical area is a high priority, it does not mean that specific evaluative research relating to any one clinical decision problem will be valuable. Similarly, proposed research to inform a particular decision in a low priority disease area may be very valuable. In this sense, attempts to identify research priorities across broad clinical areas using aggregate indicators may be mistaken. What is required is a measure of the societal value of particular research, which can inform specific clinical decisions for defined groups of patients. The additional information generated by research should be valued in a way which is consistent with the objectives and the resource constraints of health care provision. The principles of value of information analysis are briefly outlined before discussing the results of two recent opportunities to apply these methods directly to inform policy decisions about research priorities in the UK. These include a pilot study for the UK National Co-ordinating Centre for Health Technology Assessment (NCCHTA) 12 and a pilot study for the National Institute for Health and Clinical 3

4 Excellence (NICE). 13 Some of the lessons are highlighted and the methodological and policy challenges are discussed. 2. Value of Information Methods Bayesian decision theory and value of information analysis provides an analytic framework which can be used to establish the value of acquiring additional information to inform a decision problem. These methods have firm foundations in statistical decision theory 14, 15 and have been successfully used in other areas of research such as engineering and environmental risk analysis This framework values the additional information generated by further research, in a way which is consistent with the objectives and the resource constraints of heath care provision (the cost-effectiveness threshold). This allows a comparison of the potential benefits of further research with the costs of further investigation, a comparison and prioritisation of alternative research recommendations, as well as an assessment of the value of investing resources in research or other activities, such as the provision of health service. In this sense it provides a unified and coherent framework for prioritisation of research and the use of heath care technologies. 2,19,20 There is a growing literature on methods and application of value of information analysis in health technology assessment. 21,22 Below we briefly outline the principles of value of information analysis applied in the two pilot studies. 22, Expected Value of Perfect Information If the objective of a decision maker is to maximise health outcomes subject to a budget constraint, then the decision to adopt or reimburse an alternative (j) should be based on expected cost (C), expected outcomes (Q) and the cost effectiveness threshold (λ). The cost-effectiveness of an alternative j can be expressed in terms of net benefit (NB j = Q j.λ Cj). However, NB j will be uncertain and a decision must be made before it is known how the uncertain parameters in the model (θ) will resolve. The optimal decision with current information is to choose the intervention that generates the maximum expected net benefit (max j E θ NB(j, θ)). 4

5 With perfect information, decisions can be made when it is known how the uncertain parameters in the model will resolve, so different decisions can be made for different resolutions of net benefit, i.e., the decision-maker could select the intervention that maximises the net benefit for a particular value of θ (max j NB(j, θ)). However, the true values of θ are unknown, so the expected value of a decision taken with perfect information is found by averaging the maximum net benefit over the joint distribution of θ (E θ max j NB(j, θ)). The expected value of perfect information for an individual patient (or episode) is simply the difference between the expected value of the decision made with perfect information about the uncertain parameters θ, and the decision made on the basis of existing evidence: EVPI = E θ max j NB( j, θ ) max j E NB( j, θ ) However, information has public good characteristics (i.e. it is non-rival). Therefore, the EVPI for the population can be calculated based on some assessment of the effective lifetime of the technology (T), estimates of incidence over this period (I t ) and a discount rate (r): Population EVPI = I T t t t= 1 (1 + r) The EVPI for the population provides an upper bound to the value of additional research and provides a necessary condition for deciding to acquire more information, i.e., the EVPI must exceed the cost of further investigation. Similarly, where the EVPI is high, research maybe expected to have greater returns than where it is low. It should be clear that the value of information and the amount of evidence required to support a decision to adopt a technology will differ across different technologies, applied to different patient populations, and for the same technology in different circumstances (different indications, patient populations and different costeffectiveness thresholds). θ 2.1 EVPI for parameters 5

6 The EVPI surrounding the decision problem can indicate whether further research is potentially worthwhile. However, it would be useful to have an indication of what type of additional evidence would be most valuable. The value of reducing the uncertainty surrounding particular parameters in the decision model can be established using a similar approach. 22 The value of perfect information about a parameter or a subset (φ) of all the uncertain parameters θ (EVPPI) is simply the difference between the expected net benefit with perfect information about φ and the expected value with current information. With perfect information the value of φ is known and the expected net benefits are calculated over the remaining uncertainties (ψ) (max j E ψ φ NB(j, φ,ψ)), However, the true values of φ are unknown and the expected value of a decision taken with perfect information is found by averaging these maximum expected net benefits over the distribution of φ (E φ max j E ψ φ NB(j, φ,ψ)) The EVPPI for the parameter or group of parameters φ is simply the difference between the expected value of a decision made with perfect information and the expected value with current information: EVPPI ϕ = E ϕ max E NB( j, ϕ, ψ ) max E NB( j, θ ) j ψ ϕ This type of analysis can be used to focus research on the type of evidence that will be most important by identifying those parameters for which more precise estimates would be most valuable. In some circumstances, this will indicate which endpoints should be included in further experimental research. In other circumstances, it may focus research on getting more precise estimates of particular parameters that may not necessarily require experimental design and can be provided relatively quickly. This type of approach can be extended to consider the value of sample information (EVSI) which, net of the costs of sampling, provides the expected net benefit of sampling (ENBS) or the societal payoff to proposed research. Estimates of the ENBS can be used to identify optimal design of future research and the optimal portfolio of studies to inform the decision problem. 19,22,23 However, EVSI can be computationally challenging and both of the pilot studies described below were restricted to an analysis of EVPI and EVPPI. j θ 6

7 3. Recent UK policy experience Recently there have been two opportunities to apply value of information methods to directly inform policy decisions about research priorities in the UK. These are a pilot study for the UK National Co-ordinating Centre for Health Technology Assessment 12 and a pilot study for the NICE. 13 These two pilot studies are interesting for a number of reasons. Firstly they represent the first attempts to integrate this type of analysis within existing policy and decision making processes. As such they provide a test of the feasibility of conducting this analysis within the timelines demanded by policy makers and highlight some of the methodological and policy challenges. Secondly, they were conducted for organisations which either have responsibility for research commissioning or for issuing of guidance on the use of technologies but not both. The absence of combined responsibilities for both adoption decisions and research decisions poses practical difficulties for introducing more formal methods to research prioritisation, but it also highlights the dangers of making adoption decisions without considering cost of the uncertainty surrounding them. Both pilot studies comprise a series of case studies. For each pilot the application of decision analysis and value of information analysis to each case study required three core tasks to be completed: (i) the construction of a decision analytic model to represent the clinical decision problem being considered; (ii) the synthesis of evidence from a variety of sources and a probabilistic analysis to characterise the current decision uncertainty; and (iii) an estimate of the value of additional information through research to reduce decision uncertainty. 3.1 Pilot study for the NHS Health Technology Programme The UK National Co-ordinating Centre for Health Technology Assessment (NCCHTA) coordinates the UK NHS Health Technology Assessment programme which is a national programme of secondary and primary evaluative research funded as part of the Department of Health s Research and Development programme ( The NCCHTA commissioned the pilot study to consider whether 7

8 these methods might contribute to the process of identifying research priorities and commissioning research for the NHS Health Technology Assessment (HTA) programme. The NCCHTA has no role in issuing guidance on the adoption and reimbursement of health technologies and the research topics that it considers come from a variety of sources including a web-based general call for suggestions, special interest groups, as well research recommendations made by NICE. Currently these topics are prioritised by the deliberations of a number of panels and the commissioning board, based on an informal description of the topic (a vignette) and no information about cost-effectiveness or uncertainty surrounding the interventions. 3.2 Pilot study for NICE The second pilot study was commissioned by NICE. Although the primary role of NICE is to issue guidance on the use of health technologies in the NHS in England and Wales it also makes recommendations about further research as part of this guidance. Although NICE does not have the remit to require research to be conducted or to commission research itself, some assessment of whether existing evidence is sufficient to support the use of a technology, the appropriate length of time until the reconsideration of the guidance and the needs for further research must be made as part of any decision making process. The purpose of the pilot was, firstly, to address the question of what basis should these assessments made: informally by the Appraisal Committee (which formulates guidance) and the Research and Development Committee (which prioritises research recommendations) or should this assessment be informed by a more formal and transparent analysis of the value of information. The importance of decision analytic modelling for informing decisions about the use of health technologies is reflected in the recently updated NICE guide on the methods of technology appraisal. 3 One feature of the guidance is the development of reference case requirements for analysis which includes among other things a characterisation of decision uncertainty through probabilistic sensitivity analysis (PSA) 24. However, formal analysis of value of information was not specified as part of the reference case although value it was recommended. 3 So the second purpose of the pilot study was to establish the feasibility and requirements of routine value of 8

9 information analysis in addition to the existing reference case for submissions and Technology Assessment Reports. 3.3 The case studies The NCCHTA pilot study was conducted through a series of 4 case studies of research topics being considered by the relevant decision making panels and the commissioning board. They were the use of screening in age-related macular degeneration (AMD); the use of alternative manual physiotherapy techniques in asthma and in chronic obstructive pulmonary disease (COPD); and the use of alternative long-term low dose antibiotics in children with recurrent urinary tract infections (UTI). The analysis of screening for age-related macular degeneration considered the use of weekly self-screening following 1 st eye involvement with neovascular AMD. This self-screening strategy was compared to two alternatives: no screen but diagnosis and treatment of eligible AMD with photodynamic therapy following self-referral to an ophthalmologist; and a strategy of no screening and no treatment. The analysis of alternative manual chest physiotherapy interventions considered 5 alternative techniques for adults with stable COPD treated in the community and considered 4 techniques for adults treated in the community and for children treated either in the community or in hospital. The analysis of low dose antibiotics for children with recurrent UTI considered 3 low dose antibiotic regimens for infants and children, for girls and boys, and for those with and without a urinary tract abnormalities. The NICE pilot study consisted of a series of six case studies based on recent Technology Assessment Reports (TARs) which were consistent with the reference case and had already been used as the basis for recent guidance issued by NICE. These included: i) screening for age related macular degeneration (AMD); ii) glycoprotein IIb/IIIa antagonists for acute coronary syndrome (GPAs); Clopidogrel and dipyridamole in the secondary prevention of occlusive vascular events (CLO); neurominidase inhibitors for the treatment of influenza (NIs); liquid based cytology screening for cervical cancer (LBC); and beta interferon and glatiramer acetate in the management of MS (MS). The details of the models, comparators and patient groups 9

10 considered in each case study can be found in the Technology Assessment Reports published on the NICE web site and in the full report of the pilot study The Evidence In each case the analyses combined evidence from a variety of sources. The estimates of effect of the interventions were based on existing reviews of the clinical trials; meta analysis of RCT evidence (including, in some cases, indirect and mixed comparisons 25 ); as well as the results of other trial-based evaluations. Evidence about natural history and baseline event rates came from a range of sources including: epidemiological studies; pooled trial base lines; registry studies; and, in some circumstances, clinical judgement. Evidence about the quality of life for the various health states within the models was based on previously published studies as well as population surveys and trial-based analyses. Estimates of the costs of the interventions and the various clinical events were based on published trials and observational studies, as well as published sources of UK unit costs. Details of the structure of the decision models, sources of evidence, methods of synthesis and the conduct of probabilistic analysis can be found in full pilot study reports. 12,13 4. Implications for research priorities For each case study the expected value of perfect information (EVPI) was estimated for the relevant patient populations. An analysis of the value of information associated with the particular parameters within each model was also conducted (EVPPI). A summary of the results of the value of information analysis for each case study in the NCCHTA pilot and the NICE pilot are reported in table 1 and can be used to address a range of questions. 10

11 Table 1. Summary of Results for the NCCHTA and NICE pilots Case Study Patient Group EVPI EVPPI Asthma Physiotherapy Children in the community 14.5m Effect of massage on FEV 1 ( 14.2m) for Children in the community Adults in the community 0 - Children in hospital 1.2m COPD Physiotherapy Adults in the Community 0 - Recurrent UTI Prophylaxis Girls age 3, no VUR Girls age 3, VUR Girls age 1, no VUR Girls age 1, VUR Boys age 3, no VUR Boys age 3, VUR Boys age 1, no VUR Boys age 1, VUR AMD Screening Visual acuity 20/40 Glycoprotein IIb/IIIa Clopidogrel for secondary prevention Neurominidase inhibitors Liquid Based Cytology Disease modifying therapies for multiple sclerosis Visual acuity 20/80 Acute trearment following non-st-elevation acute coronary syndrome (scenario 2) Stroke Transient Ischaemic Attack Myocardial Infarction Peripheral Arterial Disease Otherwise healthy adults not at elevated risk of complications Women aged 18 to 64 years (scenario 2) Relapsing remitting and primary progressive multiple sclerosis (scenario 2) 2.241m 0.61m 0.69m 0.54m 0.04m 0.02m 0.27m 0.17m 6.18m 15.33m Effect of physical therapy on length of hospital stay ( 1.2m) The effect of prophylaxis on frequency of UTIs ( 2.24m). Long term effect is more important (>6 months 1.77m, < 6 months 0.6m). Effect of all prophylactic regimens are important (trimethoprim 1.46m, nitrofurantoin 0.61m and cotrimoxazole 1.41m). Effect of PDT on Quality of life ( 3.37m for 20/40 and 6.18m for 20/80). Progression of visual loss ( 0.19m for 20/80) 171m Relative risk of death for non acute PCI for GPA as medical management and for Clopidogrel ( 85m and 68.1m respectively). 865m 250m 710m 240m 66.7m 2.8m 86.2m Relative Risk of vascular and non vascular death ( 780m for ASA-MRdipridamole compared to clopidogrel in the stroke group) Quality of life with influenza, the effect of oselatimivir and amantadine ( 44.3m, 0.43m and 0.23m respectively) Specificity of LBC ( 1.6m) Relative risk of progression for copaxone, betaferon and rebif (22mg) ( 14m, 13.6m and 7m respectively) Costs of care costs of relapse and quality of life ( 10m, 7m and 6m respectively) 4.1 Is further research required? The value of research differed substantially across all the case studies and ranged from zero (physiotherapy for adults with either asthma or COPD treated in the community) to 865m (Clopidogrel for secondary prevention in stroke). In some cases the analysis indicated that further research should not be conducted, e.g., physiotherapy for adults asthma or COPD does not appear to be cost-effective and 11

12 any additional evidence is very unlikely to change this assessment. In other cases research should not be regarded as a priority, e.g., the EVPI surrounding LBC following the evidence from a pilot study which was previously required by NICE was low ( 2.8m scenario 2). This suggests that no additional research is needed to support the guidance to adopt LBC screening. In other cases it indicated that additional research should be regarded as a priority, e.g., the EVPI surrounding Clopidogrel for stroke patients was very high ( 865) and additional evidence should be required to support guidance. Similarly the EVPI surrounding NIs, MS and GPA was also substantially greater than the costs of further investigation ( 66.7m, 86.2m and 171m respectively). In other cases the EVPI was not as great,and these topics could be regarded as lower priorities for research. However, even in these cases further research could well be worthwhile, e.g., the EVPI for all AMD patients ( 25m), children with asthma ( 15.7m) and all children and infants with recurrent UTI ( 4.58m) may exceed the cost of further research. 4.2 What type of research? When further research appears to be worthwhile the EVPPI indicates where evidence about particular parameters will be most valuable and suggests the type of research design which might be most important. For example, the value of information for NIs was significant ( 66.7m) but it is further evidence about quality of life with influenza which is most important ( 44.3) rather than additional evidence about the effect on symptoms which would require an RCT. In the MS case study, although additional evidence about the effect on progression of the disease which would require an RCT is most valuable, evidence about cost of care and relapse, and quality of life are also valuable ( 10m and 6m respectively). In these cases further research will not require experimental design and may be less costly so these may be regarded as priorities. In other cases the analysis re-focuses the original research recommendations which were made by NICE, e.g., in the AMD case study, although further research appears to be potentially worthwhile, it is additional evidence about quality of life with and without photodynamic therapy which is valuable rather than the recommendation made by NICE of research on the performance of self screening itself. 12

13 4.3 Which sub groups? Estimates of value of information for the decision problem and for groups of parameters were also presented for relevant patient sub groups. The value of information for a particular subgroup will be determined by the size of the subgroup and the decision uncertainty for that sub group. The EVPI for sub groups can indicate whether all patient groups should be included in subsequent research or whether research can focus on particular types of patients. For example, the value of information differed across the patient groups considered in the Clopidogrel case study (from 856m to 240m) but was greater than the costs of research for every group. This suggests that further research on the stroke and MI sub groups may be regarded as a priority but evidence relevant to the TIA and PAD subgroups should also be included. The results for screening for AMD suggested that a strategy of self screening may well be cost-effective for a range of starting visual acuities. The value of additional evidence for AMD differs by visual acuity (from 6.2m to 15.3m), and suggests that additional research should not focus only on those with better starting visual acuity but should include those sub groups with lower starting visual acuities as well (AMD screening is less cost-effective for the lower visual acuity group so there is more decision uncertainty and therefore greater EVPI). The results of the UTI case study suggest that a low dose antibiotic regimen is likely to be cost-effective particularly for infants and for children with urinary tract abnormalities (VUR). There is little decision uncertainty for these groups and, therefore, limited value of information. However, for girls without urinary tract abnormalities (no VUR) there is substantial uncertainty surrounding which of the antibiotic regimens will be cost-effective. This is also the largest group and the EVPI results suggest that if further research is conducted then it is unlikely to be worthwhile unless it is targeted at non infant girls with normal urinary tracts. 4.4 Which comparators? The analysis can also indicate which comparators should be included in future research and can also suggest those that be safely excluded. For example, in the MS case study, although there is value in additional RCT evidence of the effect on 13

14 progression of the disease, it is only the effect of copaxone and betaferon which should be regarded as a priority ( 14m and 13.6m respectively). Similarly in the NIs case study there is some value in further RCTs of the effect of oselativir and amantadine on symptom days ( 0.43m and 0.23m respectively), but there is no value in further trials of zanamivir. In the UTI case study it is further evidence about the effect of prophylaxis on the frequency of UTIs which is most valuable ( 2.4m). However, it is the relative effect of two of the prophylactic regimes which is most important (trimethoprim 1.46m, cotrimoxazole 1.41m), indicating that a head to head trial of trimethoprim and cotrimoxazole maybe the priority rather than further trials comparing prophylaxis with intermittent treatment. Results of the asthma case study suggest that, out of 5 alternative physiotherapy techniques, it is only evidence of the effect of massage therapy for children treated in the community and physical therapy for children in hospital which will be valuable. 4.5 Which endpoints? Where further RCT evidence appears to be worthwhile the analysis can also indicate which endpoints should be included. For example, in the GPA case study further research is valuable and should be regarded as a priority. It also indicated that it is RCT evidence of the effect of GPA as medical management and Clopidogrel which is most valuable. However, it also indicates that it is the mortality endpoint for patients with non-acute PCI which should be the primary endpoint in any future trial. The results of the asthma case study suggest that its is additional evidence about the effect of massage on lung function for children treated in the community and the effect of physical therapy on hospital length of stay would be most valuable. 4.6 What length of follow-up In some cases it is possible to indicate the length of follow-up which may be required based on the EVPPI associated with the short term or long term effects of the intervention. For example in the UTI case study both the short term effect of prophylaxis on the frequency of UTIs (< 6 months) and the longer term effects (>6 months) were estimated separately. The results suggest that additional evidence of short term effect will not be worthwhile ( 0.6m) and, if further RCTs are conducted, 14

15 they need longer follow-up than most of the existing trials (>6 months, 1.77). In other cases it is possible to infer the length of follow-up needed based on which endpoints are valuable. For example, the GPA case study indicates that follow-up for the mortality endpoint for patients with non-acute PCI would be needed for any future trial. A more detailed discussion of the implications for research prioritisation including the implications for the design of any future research in terms of features such as the relevant patient groups and comparators, and whether experimental design was likely to be required in each of the areas, can be found in full pilot study reports. 12,13 Overall the pilot studies demonstrated that the framework of decision analysis and value of information analysis can be applied to policy-relevant decisions in a timely way. Furthermore, the framework can provide results to inform both the decision of whether a technology should be adopted based on existing evidence and whether more evidence is required to support that decision in the future. It also demonstrates that the amount and type of evidence needed to inform decisions about health technologies is essentially an empirical question. The results show that the amount of evidence required to support decisions differs across technologies with different indications and when applied to different patient groups. It shows that research priorities cannot be based simply on the current uncertainty in measures of clinical effect, e.g., physiotherapy for adults with asthma or COPD had most uncertainty about effect but also had zero EVPI (there was little decision uncertainty) where as Clopidogrel and GPAs, with substantial evidence of effect on certain endpoints, had the highest EVPI. It also demonstrates that the type of evidence required to support decisions will also differ. In some cases additional evidence of measures of effect will be most valuable and will require RCT design. In these cases it can indicate which comparators and endpoints should be included in the design and which can be safely excluded from subsequent research. In other cases, it is additional evidence about the natural history of disease or quality of life that is most valuable. It demonstrates the evidence required about a particular technology will differ across sub groups with the same indication, suggesting that effective research prioritisation should identify those groups of patients where additional evidence would be most valuable. 15

16 5. Critical issues and methodological challenges It should be recognised that the key challenges for this type of analysis are not primarily value of information analysis methods themselves but structuring decision problems, synthesis of evidence and the characterisation of uncertainty required for estimating costs and effects as well as VOI. A number of issues were highlighted by the case studies. 5.1 Structuring decision problems It is important to ensure a sufficiently wide scope for the assessment to include all the relevant alternative strategies. This includes other technologies as well as different clinical policies (e.g., start and stop criteria) for a single technology. The exclusion of alternative strategies may not change the overall guidance on use of a technology but in some cases it may have a substantial impact on the value of information and on research recommendations. For example, excluding Clopidogrel as an alternative in the GPA case study would have led to an underestimate of the EVPI. The full range of clinical policies must also be included. For example, the Clopidogrel case study was restricted to evaluating 2 year treatment policies. However, if other policies, such as life time treatment, were evaluated then, although the most cost-effective strategy may not change, the value of information may be higher and may focus on the longer term effect of secondary prevention. 5.2 Structural uncertainty Most of the case studies presented alternative scenarios to represent different views of the relevance of evidence and alternative credible assumptions about mechanisms of action. For example, a number of case studies presented scenarios to explore alternative views of the relevant evidence e.g., inclusion of related and unrelated events in the assessment of Clopidogrel, and the impact of restricting consideration of evidence at 6 months in GPAs where represented as scenarios. Others also used scenarios to explore different structural assumptions regarding the mechanism of action. For example, in the AMD case study, three scenarios of alternative structural 16

17 assumptions about how the information from self screening would change the chance of self referral to an ophthalmologist were modelled. Although the alternative assumptions had limited impact on estimates of cost-effectiveness (the overall costeffectiveness of the strategies were unchanged) they did have a more substantial impact on the value of information (from 6.3m to 30.5m for visual acuity 20/40). This suggests that evidence about the structural relationship may be as valuable as evidence about the value of the model parameters. To fully represent this uncertainty and establish the EVPPI would require a parameterisation of this uncertainty through elicitation of priors from experts and decision makers within an iterative process of analysis. 5.3 Computational expense Probabilistic analysis and, therefore, EVPI can be computationally expensive, particularly in models which require patient level simulation. The MS and LBC case studies included such models and both used different techniques to overcome the computational problems. For example, the LBC case study attempted to estimate linear relationships between model inputs and outputs, the MS case study evaluated a number of approaches including a Gaussian process which does not impose linearity. 26 Linear approximations maybe adequate for estimating costs and effect but my perform less will when estimating the value of information. The use of Gaussian process performs better than linear regression, particularly when estimating value of information, but the number of parameters which can be included is limited. It seems further work is required to pilot the use of techniques to evaluate computationally expensive models. 5.4 Evidence synthesis and characterising uncertainty Estimates of cost-effectiveness as well as value of information require the synthesis of both direct and indirect evidence for measures of effect but also for other model parameters. This was a key issue for all the case studies. The GPA case study demonstrates that only considering the 6 month trial evidence (scenario 1) would overestimate the uncertainty and value of information. The more appropriate analysis (scenario 2), which included all the trial evidence, required more sophisticated 17

18 methods of evidence synthesis with indirect comparisons. 25 The UTI case study also required indirect comparisons between the prophylactic regimens. Similarly the CLO case study employed an indirect comparison of the two main treatments of interest, Clopidogrel and modified-release dipyridamole, because no direct trial data were available. Such indirect comparisons are necessary for a full comparison of all treatment options, but are always subject to uncertainty. It is not surprising then that the relative treatment effect of Clopidogrel compared to modified-release dipyridamole on mortality, which was based on an indirect comparison, was associated with significant value of information. Of course there are alternative but credible views of the relevance and exchangeability of evidence which could be formally represented. For example, reflecting the additional uncertainty due to potential biases in the evidence, which may come from different types of study and/or suffer from publication bias. Similarly, formally modelling the exchangeability of the evidence with the parameters required in the model was generally not conducted. For example, a meta regression was conducted to establish whether relative risk was related to baseline risk in the GPA case study. The absence of strong evidence on a relationship was used to justify the combination of evidence on relative treatment effect from predominantly non-uk trials with UKspecific baseline risks. However, the uncertainty associated with this assumption was not explicitly modelled. To do so would have required formal priors to be elicited from experts. This poses the questions of which methods of elicitation and synthesis would be most appropriate 27 and who s priors should be used (the decision maker or the experts in the clinical area). It would also require an iterative process between the analyst and the decision makers which was not part of existing processes for NCCHTA or NICE and was, therefore, not available during the pilot studies. 5.5 Issues specific to value of information analysis Although the key challenges are more general and relevant to estimating cost, effect and decision uncertainty, there are a number of issues which are specific to value of information analysis where highlighted by the pilot studies. 18

19 The EVPIs presented in table 1 require estimates of the effective future population that may benefit from additional evidence. The estimates presented here are based on a time horizon of 10 years with discounting future EVI at 3.5% (see full reports for other estimates). However, the value of evidence about the current decision problem will change as technologies and prices change and other evidence becomes available. For example, the Clopidogrel case study considered only modified-release dipyridamole, as it is now considered superior to the standard release formulation. Following the technological advance which allowed an extended release formulation, modified-release dipyridamole, will likely fall out of use. In contrast, aspirin has been in use for more than the 15 years and is still recommended for use in the guidance from the Clopidogrel appraisal despite the arrival of newer, competing technologies (clopidogrel and dipyridamole). Thus there is uncertainty around the effective future lifetime of these technologies and there is likely to be different life times for different technologies relevant to the same decision problem. In the NI case study the size of the population is based on the numbers of people currently presenting to the GP with influenza like illness. But the drugs need to be used within 48 hrs of symptom onset and are likely present similarly to a common cold. Therefore true population could be much greater than estimated once the technologies are available. Whether 10 year, or other finite time horizons, are a reasonable approximations remains an open question. Modelling the impact of future changes on the value of information as well as costs and effects is complex and further work is needed to find proxies for this complex process. 28 The methods of evidence synthesis required to make comparisons between technologies generates correlation between estimates of effect. 29 For example, in the Clopidogrel case study the relative risks were correlated and it should be recognised that the EVPPIs for these relative risks individually did not account for the correlation between them and could either under or over estimate EVPI. Similarly, in the MS case study, two scenarios were used to explore the impact of regarding each of the treatments as independent or as correlated and showed the effect on the estimates of value of information. Finding linear functions of correlated parameters may offer a solution to this problem. However, in the mean time analysis should be restricted to groups of correlated parameters and if presented individually for correlated 19

20 parameters the results should be interpreted with caution (EVPPI may be under or over estimated). The value of information presented in these pilots focuses on the value of reducing uncertainty about which intervention is cost-effective assuming that those interventions regarded as cost-effective will be fully implemented (with current and with perfect information). As argued above, there is clearly an important, separate but related issue of the value of implementation, i.e. the value of ensuring that clinical practice is consistent with the current evidence of cost-effectiveness and that, if additional research is commissioned, that the findings will be implemented. The value of implementation and information can be dealt with using the same framework of analysis and can be used to inform choices between investing in research, implementation of cost-effective practice based on current evidence or investing in both research and the implementation of its findings Policy challenges The results for the NCCHTA were considered by the panels which prioritise suggested research topics and also the Commissioning Board which commissions research from the prioritised topics. In general the feedback from both was positive with the type of analysis described as interesting and potentially useful. The results of the NICE pilot were presented to the recently formed Research and Development committee who were considering which methods of prioritisation to adopt. They too in general terms regarded the analysis to be interesting and potentially useful. However, in both cases the formal analysis failed to have a significant impact on the decisions taken. For example, although the NICE Research and Development committee believed that value of information analysis should be considered and developed for future prioritisation decisions, they decided to adopt a subjective scoring system in the short term. The decisions made by the panels and commissioning board did not seem to be informed by the results of the analysis. There appears to be a number of reasons why the pilots did not have the impact on decision making that would have been hoped for. Most of those responsible for 20

21 research prioritisation were not particularly familiar with cost-effectiveness analysis, most were unfamiliar with decision modelling, and almost all had not been presented with probabilistic analysis and the type of evidence synthesis required for the analysis. This was also true at NICE since the Research and Development committee is separate from the Appraisals Committee who formulate guidance and are very familiar with these methods. As a result there was a reluctance to accept and base decisions on such unfamiliar methods. As well as a general lack of familiarity with more formal methods of evaluation, there was some questioning of the relevance and quality of some included studies as well as some of the structural assumption. Of course it would have been possible to explore the impact of these alternative views of the evidence and structural assumptions if there had been an iterative process between the decision makers and the analysts. However, in both pilots, this was not possible within the existing processes. However, the real problem may be the separation of research prioritisation and commissioning decisions from adoption and reimbursement. The increasing transparency and explicitness of the latter contrast sharply with the former. This, despite the fact that resources devoted to research and development are substantial and evidence that in some circumstances decisions about research may offer greater future benefits than the decision to adopt a technology itself. Those institutions with the remit for making adoption and reimbursement decisions come under closer public scrutiny, not least from sponsors of a technology, and have therefore been more willing to adopt transparent and explicit approaches to their decisions. Those responsible for research prioritisation and commissioning do not face the same pressure and are, therefore, able to maintain processes which are implicit and opaque. At the heart of the problem is a failure to recognise that a decision to reimburse or issue guidance on the use of a technology must also make some assessment of whether existing evidence is sufficient to support the use of a technology, the appropriate length of time until the reconsideration of the guidance and the needs for further research. Unless the adoption and research decisions are made at the same time and on the same basis then technologies will be adopted with insufficient evidence and their early diffusion will damage the prospects for gathering the evidence needed inform future clinical practice. In effect the opportunity cost of 21

22 adopting a technology without considering whether further research is needed is the value of information which maybe forgone. In some circumstances this may be greater than the net benefits offered by the technology itself. There are signs that this is being realised in the UK as NICE is now able to identify up to 3 research priorities for the NHS Health Technology Assessment programme annually, with a view to funding, If reimbursement authorities are not given the remit to commission or demand research be conducted then it may be better to deny the approval of a technology which appears cost-effective if the uncertainty and value of information is high. If not, adoption and reimbursement decisions will undermine the evidence base for future clinical practice. If research commissioners fail to adopt methods which link research to the value of decisions then they will fail to shore this up. 7. Conclusions The problems faced in conducting the pilot studies and influencing policy where not primarily technical or methodological. The real problem is the policy environment, where accountability and transparency for research prioritisation and commissioning decisions lags far behind adoption and reimbursement decisions. Until that changes it will be difficult to persuade decision makers to invest in, and to adopt more, explicit and demanding approaches. However, not so long ago decisions about the use of health technologies where equally opaque and uninformed by formal analysis. Confronting decision makers with the opportunity costs of failing to consider formal analysis made some contribution to the radical change in the policy environment, at least in the UK. The same maybe true for research decisions. Decision makers need to be confronted with the real opportunity costs of implicit and uninformed research and commissioning decisions. Equally those responsible for adoption decisions currently don t seem to take account of the uncertainty surrounding their decisions, the impact of their decisions on future research and therefore the opportunity costs of a decision to adopt a technology, i.e., the value of information forgone. The failure to consider an important, and often substantial, element of opportunity cost is leading to poor decisions and damaging the evidence base for future patients. 22