HEALTH SERVICES AND DELIVERY RESEARCH

Transcription

1 HEALTH SERVICES AND DELIVERY RESEARCH VOLUME 4 ISSUE 1 JANUARY 2016 ISSN Deveoping a methodoogica framework for organisationa case studies: a rapid review and consensus deveopment process Mark Rodgers, Sian Thomas, Meissa Harden, Giian Parker, Andrew Street and Aison Eastwood DOI /hsdr04010

2

3 Deveoping a methodoogica framework for organisationa case studies: a rapid review and consensus deveopment process Mark Rodgers, 1 * Sian Thomas, 1 Meissa Harden, 1 Giian Parker, 2 Andrew Street 3 and Aison Eastwood 1 1 Centre for Reviews and Dissemination, University of York, York, UK 2 Socia Poicy Research Unit, University of York, York, UK 3 Centre for Heath Economics, University of York, York, UK *Corresponding author Decared competing interests of authors: none Pubished January 2016 DOI: /hsdr04010 This report shoud be referenced as foows: Rodgers M, Thomas S, Harden M, Parker G, Street A, Eastwood A. Deveoping a methodoogica framework for organisationa case studies: a rapid review and consensus deveopment process. Heath Serv Deiv Res 2016;4(1).

4

5 Heath Services and Deivery Research ISSN (Print) ISSN (Onine) This journa is a member of and subscribes to the principes of the Committee on Pubication Ethics (COPE) ( Editoria contact: nihredit@southampton.ac.uk The fu HS&DR archive is freey avaiabe to view onine at Print-on-demand copies can be purchased from the report pages of the NIHR Journas Library website: Criteria for incusion in the Heath Services and Deivery Research journa Reports are pubished in Heath Services and Deivery Research (HS&DR) if (1) they have resuted from work for the HS&DR programme or programmes which preceded the HS&DR programme, and (2) they are of a sufficienty high scientific quaity as assessed by the reviewers and editors. HS&DR programme The Heath Services and Deivery Research (HS&DR) programme, part of the Nationa Institute for Heath Research (NIHR), was estabished to fund a broad range of research. It combines the strengths and contributions of two previous NIHR research programmes: the Heath Services Research (HSR) programme and the Service Deivery and Organisation (SDO) programme, which were merged in January The HS&DR programme aims to produce rigorous and reevant evidence on the quaity, access and organisation of heath services incuding costs and outcomes, as we as research on impementation. The programme wi enhance the strategic focus on research that matters to the NHS and is keen to support ambitious evauative research to improve heath services. For more information about the HS&DR programme pease visit the website: This report The research reported in this issue of the journa was funded by the HS&DR programme or one of its preceding programmes as project number 13/05/11. The contractua start date was in December The fina report began editoria review in June 2015 and was accepted for pubication in August The authors have been whoy responsibe for a data coection, anaysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors report and woud ike to thank the reviewers for their constructive comments on the fina report document. However, they do not accept iabiity for damages or osses arising from materia pubished in this report. This report presents independent research funded by the Nationa Institute for Heath Research (NIHR). The views and opinions expressed by authors in this pubication are those of the authors and do not necessariy refect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Heath. If there are verbatim quotations incuded in this pubication the views and opinions expressed by the interviewees are those of the interviewees and do not necessariy refect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Heath. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. Pubished by the NIHR Journas Library ( produced by Prepress Projects Ltd, Perth, Scotand (

6 Heath Services and Deivery Research Editor-in-Chief Professor Jo Rycroft-Maone Professor of Heath Services and Impementation Research, Bangor University, UK NIHR Journas Library Editor-in-Chief Professor Tom Waey Director, NIHR Evauation, Trias and Studies and Director of the HTA Programme, UK NIHR Journas Library Editors Professor Ken Stein Chair of HTA Editoria Board and Professor of Pubic Heath, University of Exeter Medica Schoo, UK Professor Andree Le May Chair of NIHR Journas Library Editoria Group (EME, HS&DR, PGfAR, PHR journas) Dr Martin Ashton-Key Consutant in Pubic Heath Medicine/Consutant Advisor, NETSCC, UK Professor Matthias Beck Chair in Pubic Sector Management and Subject Leader (Management Group), Queen s University Management Schoo, Queen s University Befast, UK Professor Aieen Carke Professor of Pubic Heath and Heath Services Research, Warwick Medica Schoo, University of Warwick, UK Dr Tessa Criy Director, Crysta Bue Consuting Ltd, UK Dr Peter Davidson Director of NETSCC, HTA, UK Ms Tara Lamont Scientific Advisor, NETSCC, UK Professor Eaine McCo Director, Newcaste Cinica Trias Unit, Institute of Heath and Society, Newcaste University, UK Professor Wiiam McGuire Professor of Chid Heath, Hu York Medica Schoo, University of York, UK Professor Geoffrey Meads Professor of Heath Sciences Research, Heath and Webeing Research and Deveopment Group, University of Winchester, UK Professor John Norrie Heath Services Research Unit, University of Aberdeen, UK Professor John Powe Consutant Cinica Adviser, Nationa Institute for Heath and Care Exceence (NICE), UK Professor James Raftery Professor of Heath Technoogy Assessment, Wessex Institute, Facuty of Medicine, University of Southampton, UK Dr Rob Riemsma Reviews Manager, Keijnen Systematic Reviews Ltd, UK Professor Heen Roberts Professor of Chid Heath Research, UCL Institute of Chid Heath, UK Professor Heen Snooks Professor of Heath Services Research, Institute of Life Science, Coege of Medicine, Swansea University, UK Professor Jim Thornton Professor of Obstetrics and Gynaecoogy, Facuty of Medicine and Heath Sciences, University of Nottingham, UK Pease visit the website for a ist of members of the NIHR Journas Library Board: Editoria contact: nihredit@southampton.ac.uk NIHR Journas Library

7 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Abstract Deveoping a methodoogica framework for organisationa case studies: a rapid review and consensus deveopment process Mark Rodgers, 1* Sian Thomas, 1 Meissa Harden, 1 Giian Parker, 2 Andrew Street 3 and Aison Eastwood 1 1 Centre for Reviews and Dissemination, University of York, York, UK 2 Socia Poicy Research Unit, University of York, York, UK 3 Centre for Heath Economics, University of York, York, UK *Corresponding author mark.rodgers@york.ac.uk Background: Organisationa case study proposas can be poory articuated and methodoogicay weak, raising the possibe need for pubication standards in this area. Objectives: To deveop reporting standards for organisationa case study research, with particuar appication to the UK Nationa Heath Service. Design: Rapid evidence synthesis and Dephi consensus process. Data sources: Reevant case studies and methods texts were identified through searches of ibrary cataogues, key text and author searches, focused searching of heath and socia science databases and some targeted website searching. Review methods: The reporting standards were deveoped in three stages: (1) a rapid review of the existing iterature to identify items; (2) a modified Dephi consensus process to deveop and refine content and structure; and (3) appication of the high-consensus Dephi items to two sampes of organisationa case studies to assess their feasibiity as reporting standards. Items for the Dephi consutation were identified from pubished organisationa case studies and reated methodoogica texts. Identified items were sent to a Dephi expert pane for rating over two rounds. Participants were aso asked whether or not the provisiona framework in which items were presented was appropriate, and were given the opportunity to adapt this aongside the content. In both rounds, the high-consensus threshod was set at 70% agreement among respondents for each item. High-consensus items from the Dephi consutation were then appied to previousy identified case study pubications to determine their reevance to the reporting of rea-word organisationa case studies and to better understand how the resuts of the Dephi consutation might best be impemented as a reporting standard. Resuts: One hundred and three unique reporting items were identified from 25 methodoogica texts; eight exampe case studies and 12 exempar case studies did not provide any additiona unique items. Thirteen items were utimatey rated as Shoud be reported for a organisationa case studies by at east 70% of respondents, with the degree of consensus ranging from 73% to 100%. As a whoe, exempar case studies [which had been provided by the Nationa Institute for Heath Research (NIHR) s Heath Services and Deivery Research (HSDR) programme as exampes of methodoogicay strong projects] more consistenty reported the high-consensus Dephi items than did case studies drawn from the iterature more broady. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. v

8 ABSTRACT Limitations: Time and resource constraints prevented an initia item-generation round in the Dephi consensus process. Items are therefore ikey to have been infuenced by the content, wording and assumptions of avaiabe iterature. Concusions: The high-consensus items were transated into a set of 13 reporting standards that aim to improve the consistency, rigour and reporting of organisationa case study research, thereby making it more accessibe and usefu to different audiences. The reporting standards themseves are intended primariy as a too for authors of organisationa case studies. They briefy outine broad requirements for rigorous and consistent reporting without constraining methodoogica freedom. Future work: These reporting standards shoud be incuded as part of the submission requirements for a organisationa case studies seeking funding. Though these reporting standards do not mandate specific methods, if a reporting item is not reported for egitimate methodoogica reasons, the onus is on the author to outine their rationae for the reader. Funding: The NIHR HSDR programme. vi NIHR Journas Library

9 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Contents List of tabes List of figures List of boxes Gossary List of abbreviations Pain Engish summary Scientific summary ix xi xiii xv xvii xix xxi Chapter 1 Background 1 Chapter 2 Methods 3 Research aim 3 Scope 3 Rapid review 4 Literature searching 4 Incusion and excusion criteria 6 Seection of reevant evidence 7 Data extraction 7 Synthesis 8 Dephi consensus process 8 Design 8 Participants 9 Instrumentation 9 Anaysis 9 Ethica approva 9 Transating high-consensus Dephi items into reporting standards for organisationa case studies 9 Chapter 3 Resuts 11 Rapid review 11 Rapid review: methodoogica texts 12 Rapid review: exampe case studies 14 Exempar case studies 15 Chapter 4 Dephi consensus process 17 Items identified from the iterature 17 Round 1 resuts 17 Response rate and participants 17 Rating of items 17 Round 1 comments 19 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. vii

10 CONTENTS Round 2 resuts 20 Deveopment of the questionnaire 20 Response rate and participants 21 Rating of items 22 Chapter 5 Transating high-consensus Dephi items into reporting standards for organisationa case studies 25 Exampe case studies 25 Exempar case studies 27 Generic consensus-based reporting standards for organisationa case studies 29 Chapter 6 Discussion 31 Strengths of this project 32 Weaknesses of this project 33 Chapter 7 Concusions 35 Key concusions from the project 35 Impications for research 35 Impications for practice 36 Acknowedgements 37 References 39 Appendix 1 Search strategies 45 Appendix 2 Synthesised data extraction of methodoogica texts 51 Appendix 3 Dephi consensus process round 1 questionnaire 63 Appendix 4 Dephi consensus process round 2 questionnaire 89 Appendix 5 Respondent comments from round Appendix 6 Compete respondent comments from round Appendix 7 Items with 70% consensus in round Appendix 8 Items with no overa consensus 139 viii NIHR Journas Library

11 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 List of tabes TABLE A Consensus standards for the reporting of organisationa case studies xxiii TABLE 1 List of methodoogica texts 13 TABLE 2 Exampe case studies 14 TABLE 3 Exempar case studies funded by the NIHR HSDR programme 15 TABLE 4 Characteristics of respondents 18 TABLE 5 Items identified as essentia by > 70% of respondents in round 1 20 TABLE 6 Round 2 respondent opinions on the vaue and feasibiity of reporting standards for organisationa case studies 22 TABLE 7 High-consensus items identified in round 2 23 TABLE 8 Items meeting 70% consensus ony when a and specific categories are combined 24 TABLE 9 Assessing exampe case studies against the high-consensus Dephi items 26 TABLE 10 Assessing the HSDR programme exempar case studies against the high-consensus Dephi items 28 TABLE 11 Consensus standards for the reporting of organisationa case studies 30 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. ix

12

13 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 List of figures FIGURE 1 Outine of the research process 3 FIGURE 2 A Preferred Reporting Items for Systematic Reviews and Meta-Anayses (PRISMA) fow chart 11 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xi

14

15 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 List of boxes BOX 1 Seected key pubications from authors featuring prominenty in searches 5 BOX 2 Research interests of respondents (round 1 and round 2) 19 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xiii

16

17 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Gossary Case study A method of research that engages in the cose, detaied examination of a singe exampe or phenomenon. Organisationa case study A case study reating to an organised body of peope with a particuar purpose, such as a business, government department or charity group. Paradigm A genera set of assumptions, questions and methods that structures a fied of inquiry at any given time. Positivism A phiosophica and socia scientific doctrine that uphods the primacy of sense experience and empirica evidence as the basis for knowedge and research. Post structuraism Attributes subjectivity and meaning to systems of differentia reations, such as anguage or power; beyond that, it seeks to expain the generation of those structures, either in terms of historica change or of deeper inguistic and experientia reaities. Reativism A phiosophica position that a points of view are equay vaid, and that a truth is reative to the individua. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xv

18

19 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 List of abbreviations HSDR NIHR Heath Services and Deivery Research Nationa Institute for Heath Research PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Anayses Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xvii

20

21 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Pain Engish summary Organisationa case studies typicay invove the cose, detaied examination of an organised body of peope with a particuar purpose, such as a business, government department or charity group. This research method can be used to understand activity and behaviour within a rea-ife context. However, the actua methods proposed for organisationa case studies are often not ceary described. We aimed to deveop a set of reporting standards for organisationa case study research reevant to the UK NHS. Reporting standards are a ist of criteria that can be used to improve the consistency, rigour and reporting of research, making it more accessibe and usefu to different audiences. We deveoped the standards by first coecting together practica information about reporting from pubished research methods texts and reevant organisationa case studies. We presented this information to an onine pane of experts as a ist of items. We reduced and refined these items over two rounds, unti ony those agreed to be essentia for the reporting of organisationa case studies remained. Utimatey, there was agreement among experts on 13 items, which together formed the basis of the reporting standards. Appication of these standards to existing organisationa case studies suggested that they can be used to improve the consistency, rigour and reporting of future research. We suggest that the reporting standards be further tested (and possiby refined) for use by researchers seeking pubic funding. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xix

22

23 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Scientific summary Background Case study is commony understood to be a method of research that engages in the cose, detaied examination of a singe exampe or phenomenon, and is an approach commony used to understand activity and behaviour within a rea-ife context. Organisationa case studies are concerned with an organised body of peope with a particuar purpose, such as a business, government department or charity group. When conducted we, organisationa case studies can provide insights into organisationa changes in heath care that are not easiy achieved through other study designs. They can be used to identify faciitators and barriers to the deivery of services and to hep understand the infuence of context; high-quaity organisationa case studies have been used to examine ways of working in acute care, primary care, menta heath services, residentia care and across the NHS more broady. Athough good-quaity studies wi be funded and pubished, some organisationa case study proposas submitted to the Nationa Institute for Heath Research (NIHR) s Heath Services and Deivery Research (HSDR) programme can be poory articuated and methodoogicay weak, raising the possibe need for pubication standards in this area. Reporting standards aready exist for a range of study designs, incuding randomised trias, observationa studies, systematic reviews, cinica case reports, quaitative research, reaist syntheses, meta-narrative reviews, diagnostic/prognostic studies, quaity improvement studies and economic evauations. However, a search of the EQUATOR (Enhancing the QUAity and Transparency Of heath Research) cearinghouse for reporting guideines suggests that to date no such standards have been reported for organisationa case studies. Objectives To deveop reporting standards for organisationa case study research, with particuar appication to the UK NHS. Methods The reporting standards were deveoped in three stages: 1. a rapid review of the existing iterature to identify content 2. a modified Dephi consensus process to deveop and refine content and structure 3. appication of the high-consensus Dephi items to two sampes of organisationa case studies to assess their feasibiity as reporting standards. Data sources Reevant case studies and methods texts were identified through searches of ibrary cataogues, key author searches, focused searching of heath and socia science databases and some targeted website searching. Participants Experts and parties interested in the conduct of organisationa case study research (methodoogists, research funders, journa editors, interested poicy-makers and practitioners) were approached to participate. Individuas were identified through the rapid review, persona contacts, and by contacting the foowing organisations: Heath Services Research Network, the Socia Research Association, the UK Evauation Society and the Nationa Centre for Research Methods. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xxi

24 SCIENTIFIC SUMMARY Review methods Items for the Dephi were identified from the foowing texts: organisationa case studies reating to an organised body of peope with a particuar purpose, such as a business, government department or charity group, identified from searches or from case study projects considered by HSDR as being of high quaity methodoogica texts providing practica advice specific to the conduct of organisationa case study research. Two reviewers independenty screened tites and abstracts, with disagreements resoved through discussion or consutation with a third reviewer. Data extraction was conducted by one reviewer and checked by a second. Extracted items were dedupicated and cassified into a provisiona framework: panning and study design data coection data anaysis reporting. Dephi consensus methods The provisiona framework and its constituent items were sent to the Dephi expert pane for rating. The Dephi consisted of two rounds: In the first round, participants were presented with a the unique items identified from the rapid review. They were asked to rate each item as being essentia, desirabe or not necessary for the reporting of organisationa case studies. Participants were aso asked whether or not the provisiona framework (grouping items into panning/design, data coection, anaysis and reporting) in which items were presented was appropriate, and were given the opportunity to adapt this aongside the content. In the second round, participants received a restructured ist of items incorporating feedback from the resuts of the first round. They were given the opportunity to identify the reporting items as being reevant to a, some or no organisationa case studies. In both rounds, the high-consensus threshod was set at 70% agreement among respondents for each item. The ist of items with high consensus after the second round was appied to previousy identified case study pubications in order to (1) determine the reevance of these items to the reporting of rea-word organisationa case studies and (2) better understand how the resuts of the Dephi consutation might best be impemented as a reporting standard. Resuts An initia poo of 103 unique reporting items was identified from 25 methodoogica texts; eight exampe case studies (17 pubications) and 12 exempar case studies, which had been provided by the HSDR programme as exampes of methodoogicay strong projects (16 pubications), did not provide any additiona unique items. Of 36 experts invited to take part in the Dephi consensus process, 19 (53%) responded to the first round invitation. Fifteen respondents competed the entire round 2 questionnaire, 14 of whom had aso taken part in the first round. The majority of respondents in round 1 were researchers (80%) with substantia experience of authoring or otherwise contributing to organisationa case study research. In the first round, 10 items met the predefined minimum 70% agreement eve for being essentia, with consensus ranging from 74% to 95%. xxii NIHR Journas Library

25 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 In the second round, a sighty greater proportion of respondents thought a reporting standard for reporting organisationa case studies was desirabe than did not, though severa were uncertain. Others suggested that the usefuness of any standards woud depend upon how and where they are appied. Respondents were simiary divided about whether or not a reporting standard woud be feasibe for organisationa case studies. Thirteen items were utimatey rated as shoud be reported for a organisationa case studies by at east 70% of respondents, with the degree of consensus ranging from 73% to 100%. As a whoe, exempar case studies considered methodoogicay strong by the HSDR programme more consistenty reported the high-consensus Dephi items than did case studies drawn from iterature more broady. Of eeven exempar pubications, six (55%) reported a 13 items, compared with just 3 out of 17 (18%) of the exampe organisationa case study pubications. The high-consensus items were transated into a set of 13 reporting standards grouped into four sections: describing the design describing the data coection describing the data anaysis interpreting the resuts. TABLE A Consensus standards for the reporting of organisationa case studies Reporting item Describing the design 1. Define the research as a case study 2. State the broad aims of the study 3. State the research question(s)/hypotheses 4. Identify the specific case(s) and justify the seection Describing the data coection 5. Describe how data were coected 6. Describe the sources of evidence used 7. Describe any ethica considerations and obtainment of reevant approvas, access and permissions Describing the data anaysis 8. Describe the anaysis methods Interpreting the resuts 9. Describe any inherent shortcomings in the design and anaysis and how these might have infuenced the findings 10. Consider the appropriateness of methods used for the question and subject matter and why it was that quaitative methods were appropriate 11. Discuss the data anaysis 12. Ensure that the assertions are sound, neither over- nor under-interpreting the data 13. State any caveats about the study Page number on which item was reported Page number of justification for not reporting Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xxiii

26 SCIENTIFIC SUMMARY Concusions These reporting standards aim to improve the consistency, rigour and reporting of organisationa case study research, thereby making it more accessibe and usefu to different audiences. These audiences incude research sponsors, who need to make decisions about whether to fund proposed case studies; ethics and research advisory groups, who require carity about the specific panned methods; peer reviewers, who need to be abe to evauate the robustness of a competed case study; and readers and poicy-makers, who need to understand how the findings of an organisationa case study might be interpreted and impemented. The reporting standards themseves are intended primariy as a too for authors of organisationa case studies. They briefy outine broad requirements for rigorous and consistent reporting, without constraining methodoogica freedom. Impemented propery, these shoud faciitate peer review of organisationa case studies and give greater confidence to the readers of this kind of research. Impications for research These reporting standards shoud be incuded as part of the submission requirements for a organisationa case studies seeking funding. Though these reporting standards do not mandate specific methods, if a reporting standard is not reported for egitimate methodoogica reasons, the onus is on the author to outine their rationae for the reader. Fina report manuscripts shoud be accompanied by a version of the reporting standards competed by the study author(s), and both documents shoud be made avaiabe to peer reviewers. Funding boards may want to coect feedback from users (incuding commissioners, authors, peer reviewers) in order to buid engagement with the concept of reporting standards for organisationa case studies and to coect evidence that coud be used to evauate and/or further refine these standards. Funding Funding for this study was provided by the HSDR programme of the NIHR. xxiv NIHR Journas Library

27 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Chapter 1 Background Case study is commony understood to be a method of research that engages in the cose, detaied examination of a singe exampe or phenomenon, and is an approach commony used to understand activity and behaviour within a rea-ife context. When conducted we, organisationa case studies can provide insights into organisationa changes in heath care that are not easiy achieved through other study designs. They can be used to identify faciitators and barriers to the deivery of services and to hep understand the infuence of context, and high-quaity organisationa case studies have been used to examine ways of working in acute care, 1 primary care, 2 menta heath services, 3 residentia care 4 and across the NHS more broady. 5 7 Yin 8 describes a case study to be the preferred research method when (1) the main research questions are how? or why? ; (2) a researcher has itte or no contro over behavioura events; and (3) the focus of the study is a contemporary (as opposed to historica) phenomenon. However, there is no set methodoogy for a case study and the term is often used oosey, but typicay combines quaitative and quantitative data coection with a strong observationa component. Case study research can be conducted from both reativist and positivist perspectives, and can be used to generate new theories, vaidate existing theories or address both of these matters. 9 An individua case can be studied aone to understand something about the case itsef and its contexts, or compared with other cases for iustrative, expanatory or evauative purposes. 10 The case study has been proposed as an appropriate method for describing, expaining, predicting or controing processes associated with phenomena at the individua, group or organisationa eve. 11 The majority of the Nationa Institute for Heath Research (NIHR) s Heath Services and Deivery Research (HSDR) programme-funded case studies are specificay concerned with description or expanation at the organisationa eve. In the past, many proposas for organisationa case studies submitted to the HSDR programme have been poory articuated and methodoogicay weak and were therefore unikey to deiver robust research findings. Specific areas of concern raised by the HSDR programme incuded: Absence of cear research questions that the case study method is intended to answer. Vagueness about samping frame/strategy. Proposas where it is not cear how organisations or sites were seected or what was the basis for samping. Insufficient theoretica basis. Many studies ack an organising theoretica framework; this can affect a stages, from samping of sites through to anaysis and how findings can add to the body of knowedge. Lack of carity about the unit of anaysis. Some weaker proposas wi not identify the unit of interest, whereas good case studies may incude data streams around the individua, team, organisation and wider system, and wi be expicit about the overa study design and interest. Lack of any cear pans for anaysis. Some proposas make no attempt to ook activey for data that chaenge emerging theories, findings or knowedge of systematic comparative case anayses. Many such studies are purey descriptive, without any expanatory power. Lack of carity about how data from a range of sources wi be integrated. Proposas increasingy caim to use reaist evauation methods for case study work, but make no attempt to estabish a programme theory, identify candidate mechanisms or describe other features of reaist evauation. Consequenty, the HSDR programme expressed an interest in identifying the characteristics of good-quaity case study research, and in devising quaity and pubication standards, with particuar appication to the NHS. More specificay, they described the need for a rapid evidence review aongside a Dephi or expert consensus-buiding exercise to identify eements of good practice and standards for reporting and pubication. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 1

28 BACKGROUND Athough some authors have proposed practica methodoogica guideines for case study research methods, 8,9 these have not been universay adopted. The broad diversity of approaches used within organisationa case studies and the contrasting paradigms that underpin these approaches mean that any attempt to deveop definitive methodoogica guidance in this area is ikey to be both highy contentious and resource-intensive. However, the abiity of research funders, peer reviewers and other research users to estabish methodoogica quaity is at east party contingent on the carity of expanation of the methods proposed or utiised. Indeed, severa of the concerns raised by the HSDR programme above specificay refer to vagueness or ack of carity around the reporting of proposed research methods. Reporting standards aready exist for a range of study designs, incuding randomised trias, 12 observationa studies, 13 systematic reviews, 14 cinica case reports, 15 quaitative research, 16 reaist syntheses, 17 meta-narrative reviews, 18 diagnostic/prognostic studies, 19 quaity improvement studies 20 and economic evauations. 21 However, to date, no such standards have been reported for organisationa case studies. By encouraging authors to consider how their methods are presented, the avaiabiity of an appropriate set of reporting standards for organisationa case studies aso has the potentia to improve research conduct in genera. A suitabe first step towards better conduct of organisationa case studies woud be to estabish agreement about what needs to be reported among the diverse group of researchers who undertake this kind of research. Shoud further guidance be needed about appropriate methods, the reporting standards can be used as a foundation on which to buid. The aim of this project has been to identify the characteristics of good-quaity organisationa case study research and devise reporting standards, with particuar appication to the NHS. Athough a range of opinions and experiences has been sought, the project has not been concerned with case studies outside the remit of the work funded by the HSDR programme. Therefore, it is not intended the reporting standards shoud be appied to case studies of individuas or to those conducted in other research fieds. In the first instance, we woud anticipate that these standards shoud be used to improve the standard of submissions to the HSDR programme. There may be further potentia for dissemination of the standards to the wider word of organisationa case study researchers. 2 NIHR Journas Library

29 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Chapter 2 Methods Research aim The aim of the project was to deveop reporting standards for organisationa case study research, with particuar appication to the UK Nationa Heath Service. Scope We deveoped the reporting standards in three stages, as shown in Figure 1: 1. a rapid review of the existing iterature to identify content for the standards 2. a Dephi consensus process to deveop and refine the fina set of standards 3. appication of the high-consensus Dephi items to two sampes of organisationa case studies to assess their feasibiity as reporting standards. Literature search Methods texts describing the purpose, design and conduct of organisationa case studies Exampe case studies reevant to the NHS Exempar case studies 1. Rapid review Item generation and provisiona framework 2. Consensus process Deveopment and refinement of framework and content Transating high-consensus Dephi items into reporting standards 3. Feasibiity assessment FIGURE 1 Outine of the research process. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 3

30 METHODS Rapid review A rapid review was used to generate items to popuate a provisiona framework for organisationa case studies. Systematic review methodoogy was used to identify artices and extract and synthesise data. Because of the rapid nature of the review, the process was ess exhaustive and contained ess detai than woud have been achievabe from a fu systematic review. Literature searching The aim of the search strategy was to identify materia about organisationa case study methods. It was anticipated that the iterature on this topic woud be found in textbooks, book chapters, journa artices and research methods guidance; therefore, the search strategy consisted of searches of ibrary cataogues, key author searches, focused searching of heath and socia science databases and some targeted website searching. Library cataogue searches The foowing ibrary cataogues speciaising in heath management iterature were searched to ocate books on case study methods: Heath Services Management Centre ONLINE (via the University of Birmingham; faciities/hsmc-ibrary/ibrary-resources/index.aspx) Heath Management Onine (via NHS Scotand; The King s Fund Library Database ( Key author searches Five authors, detaied in Box 1, featured prominenty in the initia iterature searches: David Byrne, Bent Fyvbjerg, Roger Gomm, Chares Ragin and Robert K Yin. Searches were carried out via Googe (Googe Inc., Mountain View, CA, USA) on each author to ocate any ists of their pubications. Pubication ists were found for each author on either their institution website or, where this was not avaiabe, through searches of Googe Books and Googe Schoar. Database searches Initia database test searches reveaed difficuties in ocating case study methodoogy iterature efficienty without retrieving arge numbers of irreevant resuts. Therefore, as this was a rapid review, a highy focused search strategy was deveoped on MEDLINE (via Ovid) to identify papers about organisationa case study methods. Focusing of subject headings, use of subheadings and searching in the tite-ony fied were utiised in the strategy. Searches were restricted to Engish-anguage papers. A more imited range of databases than usua for a fu systematic review was searched. In particuar, no specific databases of conference proceedings, theses or foreign-anguage studies were searched. Reevant databases covering iterature from heath, heath management and socia science were searched: MEDLINE & MEDLINE In-Process, Appied Socia Sciences Index and Abstracts (ASSIA), Heath Management Information Consortium, PsycINFO and the Socia Science Citation Index. The MEDLINE strategy was adapted for use in each database. Website searches The foowing websites were searched to identify any guidance documents on case study methods: ESRC Nationa Centre for Research Methods ( ESRC Research Methods Programme ( The Socia Research Association ( Methods@Manchester ( 4 NIHR Journas Library

31 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 BOX 1 Seected key pubications from authors featuring prominenty in searches David Byrne, Socia Scientist, Schoo of Appied Socia Sciences, University of Durham Byrne D. Compex reaist and configurationa approaches to cases: a radica synthesis. In Byrne D, Ragin C, editors. The SAGE Handbook of Case-based Methods. London: SAGE Pubications; pp Byrne D. Introduction: case-based methods: why we need them; what they are; how to do them. In Byrne D, Ragin C, editors. The SAGE Handbook of Case-based Methods. London: SAGE Pubications; pp Byrne DS. Compexity, configuration and cases. Theor Cut Soc 2005;22: Bent Fyvbjerg, Economic Geographer, Said Business Schoo, Oxford University Fyvbjerg B, Landman T, Schram S. Important next steps in phronetic socia science. In Fyvbjerg B, Landman T, Schram S, editors. Rea Socia Science: Appied Phronesis. Cambridge: Cambridge University Press; pp Fyvbjerg B, Landman T, Schram S. Rea Socia Science: Appied Phronesis. Cambridge: Cambridge University Press; Fyvbjerg B. Case study. In Denzin NK, Lincon YS, editors. The SAGE Handbook of Quaitative Research. 4th edn. Thousand Oaks, CA: SAGE Pubications; pp Roger Gomm, Socia Scientist, Schoo of Heath and Socia Wefare, The Open University Gomm R. Key Concepts in Socia Research Methods. London: Pagrave Macmian; Gomm R, Davies C. Using Evidence in Heath and Socia Care. London: SAGE Pubications; Gomm R, Hammersey M, Foster P. Case Study Method Key Issues, Key Texts. London: SAGE Pubications; Chares Ragin, Socioogist, Schoo if Socioogy, University of Caifornia at Irvine Ragin CC. Refections on casing and case-oriented research. In Byrne D, Ragin CC, editors. The SAGE Handbook of Case-based Methods. London: SAGE Pubications; pp Ragin CC, Schneider G. Case-oriented theory buiding and theory testing. In Wiiams M, Vogt P, editors. The SAGE Handbook of Methodoogica Innovations. London: SAGE Pubications; pp Ragin CC, Schneider G. Comparative poitica anaysis: six case-oriented strategies. In Amenta E, Nash K, Scott A, editors. The New Backwe Companion to Poitica Socioogy. Chichester: Backwe; pp Robert K Yin, socia scientist, COSMOS Corporation Yin RK. Appications of Case Study Research. 3rd edn. London: SAGE Pubications; Yin RK. Case study methods. In Cooper H, editor. APA Handbook of Research Methods in Psychoogy, Vo 2: Research Designs: Quantitative, Quaitative, Neuropsychoogica, and Bioogica. Washington, DC: American Psychoogica Association; pp Yin RK. Case Study Research: Design and Methods. Thousand Oaks, CA: SAGE Pubications; Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 5