HEALTH SERVICES AND DELIVERY RESEARCH

Transcription

1 HEALTH SERVICES AND DELIVERY RESEARCH VOLUME 4 ISSUE 1 JANUARY 2016 ISSN Deveoping a methodoogica framework for organisationa case studies: a rapid review and consensus deveopment process Mark Rodgers, Sian Thomas, Meissa Harden, Giian Parker, Andrew Street and Aison Eastwood DOI /hsdr04010

2

3 Deveoping a methodoogica framework for organisationa case studies: a rapid review and consensus deveopment process Mark Rodgers, 1 * Sian Thomas, 1 Meissa Harden, 1 Giian Parker, 2 Andrew Street 3 and Aison Eastwood 1 1 Centre for Reviews and Dissemination, University of York, York, UK 2 Socia Poicy Research Unit, University of York, York, UK 3 Centre for Heath Economics, University of York, York, UK *Corresponding author Decared competing interests of authors: none Pubished January 2016 DOI: /hsdr04010 This report shoud be referenced as foows: Rodgers M, Thomas S, Harden M, Parker G, Street A, Eastwood A. Deveoping a methodoogica framework for organisationa case studies: a rapid review and consensus deveopment process. Heath Serv Deiv Res 2016;4(1).

4

5 Heath Services and Deivery Research ISSN (Print) ISSN (Onine) This journa is a member of and subscribes to the principes of the Committee on Pubication Ethics (COPE) ( Editoria contact: [email protected] The fu HS&DR archive is freey avaiabe to view onine at Print-on-demand copies can be purchased from the report pages of the NIHR Journas Library website: Criteria for incusion in the Heath Services and Deivery Research journa Reports are pubished in Heath Services and Deivery Research (HS&DR) if (1) they have resuted from work for the HS&DR programme or programmes which preceded the HS&DR programme, and (2) they are of a sufficienty high scientific quaity as assessed by the reviewers and editors. HS&DR programme The Heath Services and Deivery Research (HS&DR) programme, part of the Nationa Institute for Heath Research (NIHR), was estabished to fund a broad range of research. It combines the strengths and contributions of two previous NIHR research programmes: the Heath Services Research (HSR) programme and the Service Deivery and Organisation (SDO) programme, which were merged in January The HS&DR programme aims to produce rigorous and reevant evidence on the quaity, access and organisation of heath services incuding costs and outcomes, as we as research on impementation. The programme wi enhance the strategic focus on research that matters to the NHS and is keen to support ambitious evauative research to improve heath services. For more information about the HS&DR programme pease visit the website: This report The research reported in this issue of the journa was funded by the HS&DR programme or one of its preceding programmes as project number 13/05/11. The contractua start date was in December The fina report began editoria review in June 2015 and was accepted for pubication in August The authors have been whoy responsibe for a data coection, anaysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors report and woud ike to thank the reviewers for their constructive comments on the fina report document. However, they do not accept iabiity for damages or osses arising from materia pubished in this report. This report presents independent research funded by the Nationa Institute for Heath Research (NIHR). The views and opinions expressed by authors in this pubication are those of the authors and do not necessariy refect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Heath. If there are verbatim quotations incuded in this pubication the views and opinions expressed by the interviewees are those of the interviewees and do not necessariy refect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Heath. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. Pubished by the NIHR Journas Library ( produced by Prepress Projects Ltd, Perth, Scotand (

6 Heath Services and Deivery Research Editor-in-Chief Professor Jo Rycroft-Maone Professor of Heath Services and Impementation Research, Bangor University, UK NIHR Journas Library Editor-in-Chief Professor Tom Waey Director, NIHR Evauation, Trias and Studies and Director of the HTA Programme, UK NIHR Journas Library Editors Professor Ken Stein Chair of HTA Editoria Board and Professor of Pubic Heath, University of Exeter Medica Schoo, UK Professor Andree Le May Chair of NIHR Journas Library Editoria Group (EME, HS&DR, PGfAR, PHR journas) Dr Martin Ashton-Key Consutant in Pubic Heath Medicine/Consutant Advisor, NETSCC, UK Professor Matthias Beck Chair in Pubic Sector Management and Subject Leader (Management Group), Queen s University Management Schoo, Queen s University Befast, UK Professor Aieen Carke Professor of Pubic Heath and Heath Services Research, Warwick Medica Schoo, University of Warwick, UK Dr Tessa Criy Director, Crysta Bue Consuting Ltd, UK Dr Peter Davidson Director of NETSCC, HTA, UK Ms Tara Lamont Scientific Advisor, NETSCC, UK Professor Eaine McCo Director, Newcaste Cinica Trias Unit, Institute of Heath and Society, Newcaste University, UK Professor Wiiam McGuire Professor of Chid Heath, Hu York Medica Schoo, University of York, UK Professor Geoffrey Meads Professor of Heath Sciences Research, Heath and Webeing Research and Deveopment Group, University of Winchester, UK Professor John Norrie Heath Services Research Unit, University of Aberdeen, UK Professor John Powe Consutant Cinica Adviser, Nationa Institute for Heath and Care Exceence (NICE), UK Professor James Raftery Professor of Heath Technoogy Assessment, Wessex Institute, Facuty of Medicine, University of Southampton, UK Dr Rob Riemsma Reviews Manager, Keijnen Systematic Reviews Ltd, UK Professor Heen Roberts Professor of Chid Heath Research, UCL Institute of Chid Heath, UK Professor Heen Snooks Professor of Heath Services Research, Institute of Life Science, Coege of Medicine, Swansea University, UK Professor Jim Thornton Professor of Obstetrics and Gynaecoogy, Facuty of Medicine and Heath Sciences, University of Nottingham, UK Pease visit the website for a ist of members of the NIHR Journas Library Board: Editoria contact: [email protected] NIHR Journas Library

7 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Abstract Deveoping a methodoogica framework for organisationa case studies: a rapid review and consensus deveopment process Mark Rodgers, 1* Sian Thomas, 1 Meissa Harden, 1 Giian Parker, 2 Andrew Street 3 and Aison Eastwood 1 1 Centre for Reviews and Dissemination, University of York, York, UK 2 Socia Poicy Research Unit, University of York, York, UK 3 Centre for Heath Economics, University of York, York, UK *Corresponding author [email protected] Background: Organisationa case study proposas can be poory articuated and methodoogicay weak, raising the possibe need for pubication standards in this area. Objectives: To deveop reporting standards for organisationa case study research, with particuar appication to the UK Nationa Heath Service. Design: Rapid evidence synthesis and Dephi consensus process. Data sources: Reevant case studies and methods texts were identified through searches of ibrary cataogues, key text and author searches, focused searching of heath and socia science databases and some targeted website searching. Review methods: The reporting standards were deveoped in three stages: (1) a rapid review of the existing iterature to identify items; (2) a modified Dephi consensus process to deveop and refine content and structure; and (3) appication of the high-consensus Dephi items to two sampes of organisationa case studies to assess their feasibiity as reporting standards. Items for the Dephi consutation were identified from pubished organisationa case studies and reated methodoogica texts. Identified items were sent to a Dephi expert pane for rating over two rounds. Participants were aso asked whether or not the provisiona framework in which items were presented was appropriate, and were given the opportunity to adapt this aongside the content. In both rounds, the high-consensus threshod was set at 70% agreement among respondents for each item. High-consensus items from the Dephi consutation were then appied to previousy identified case study pubications to determine their reevance to the reporting of rea-word organisationa case studies and to better understand how the resuts of the Dephi consutation might best be impemented as a reporting standard. Resuts: One hundred and three unique reporting items were identified from 25 methodoogica texts; eight exampe case studies and 12 exempar case studies did not provide any additiona unique items. Thirteen items were utimatey rated as Shoud be reported for a organisationa case studies by at east 70% of respondents, with the degree of consensus ranging from 73% to 100%. As a whoe, exempar case studies [which had been provided by the Nationa Institute for Heath Research (NIHR) s Heath Services and Deivery Research (HSDR) programme as exampes of methodoogicay strong projects] more consistenty reported the high-consensus Dephi items than did case studies drawn from the iterature more broady. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. v

8 ABSTRACT Limitations: Time and resource constraints prevented an initia item-generation round in the Dephi consensus process. Items are therefore ikey to have been infuenced by the content, wording and assumptions of avaiabe iterature. Concusions: The high-consensus items were transated into a set of 13 reporting standards that aim to improve the consistency, rigour and reporting of organisationa case study research, thereby making it more accessibe and usefu to different audiences. The reporting standards themseves are intended primariy as a too for authors of organisationa case studies. They briefy outine broad requirements for rigorous and consistent reporting without constraining methodoogica freedom. Future work: These reporting standards shoud be incuded as part of the submission requirements for a organisationa case studies seeking funding. Though these reporting standards do not mandate specific methods, if a reporting item is not reported for egitimate methodoogica reasons, the onus is on the author to outine their rationae for the reader. Funding: The NIHR HSDR programme. vi NIHR Journas Library

9 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Contents List of tabes List of figures List of boxes Gossary List of abbreviations Pain Engish summary Scientific summary ix xi xiii xv xvii xix xxi Chapter 1 Background 1 Chapter 2 Methods 3 Research aim 3 Scope 3 Rapid review 4 Literature searching 4 Incusion and excusion criteria 6 Seection of reevant evidence 7 Data extraction 7 Synthesis 8 Dephi consensus process 8 Design 8 Participants 9 Instrumentation 9 Anaysis 9 Ethica approva 9 Transating high-consensus Dephi items into reporting standards for organisationa case studies 9 Chapter 3 Resuts 11 Rapid review 11 Rapid review: methodoogica texts 12 Rapid review: exampe case studies 14 Exempar case studies 15 Chapter 4 Dephi consensus process 17 Items identified from the iterature 17 Round 1 resuts 17 Response rate and participants 17 Rating of items 17 Round 1 comments 19 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. vii

10 CONTENTS Round 2 resuts 20 Deveopment of the questionnaire 20 Response rate and participants 21 Rating of items 22 Chapter 5 Transating high-consensus Dephi items into reporting standards for organisationa case studies 25 Exampe case studies 25 Exempar case studies 27 Generic consensus-based reporting standards for organisationa case studies 29 Chapter 6 Discussion 31 Strengths of this project 32 Weaknesses of this project 33 Chapter 7 Concusions 35 Key concusions from the project 35 Impications for research 35 Impications for practice 36 Acknowedgements 37 References 39 Appendix 1 Search strategies 45 Appendix 2 Synthesised data extraction of methodoogica texts 51 Appendix 3 Dephi consensus process round 1 questionnaire 63 Appendix 4 Dephi consensus process round 2 questionnaire 89 Appendix 5 Respondent comments from round Appendix 6 Compete respondent comments from round Appendix 7 Items with 70% consensus in round Appendix 8 Items with no overa consensus 139 viii NIHR Journas Library

11 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 List of tabes TABLE A Consensus standards for the reporting of organisationa case studies xxiii TABLE 1 List of methodoogica texts 13 TABLE 2 Exampe case studies 14 TABLE 3 Exempar case studies funded by the NIHR HSDR programme 15 TABLE 4 Characteristics of respondents 18 TABLE 5 Items identified as essentia by > 70% of respondents in round 1 20 TABLE 6 Round 2 respondent opinions on the vaue and feasibiity of reporting standards for organisationa case studies 22 TABLE 7 High-consensus items identified in round 2 23 TABLE 8 Items meeting 70% consensus ony when a and specific categories are combined 24 TABLE 9 Assessing exampe case studies against the high-consensus Dephi items 26 TABLE 10 Assessing the HSDR programme exempar case studies against the high-consensus Dephi items 28 TABLE 11 Consensus standards for the reporting of organisationa case studies 30 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. ix

12

13 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 List of figures FIGURE 1 Outine of the research process 3 FIGURE 2 A Preferred Reporting Items for Systematic Reviews and Meta-Anayses (PRISMA) fow chart 11 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xi

14

15 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 List of boxes BOX 1 Seected key pubications from authors featuring prominenty in searches 5 BOX 2 Research interests of respondents (round 1 and round 2) 19 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xiii

16

17 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Gossary Case study A method of research that engages in the cose, detaied examination of a singe exampe or phenomenon. Organisationa case study A case study reating to an organised body of peope with a particuar purpose, such as a business, government department or charity group. Paradigm A genera set of assumptions, questions and methods that structures a fied of inquiry at any given time. Positivism A phiosophica and socia scientific doctrine that uphods the primacy of sense experience and empirica evidence as the basis for knowedge and research. Post structuraism Attributes subjectivity and meaning to systems of differentia reations, such as anguage or power; beyond that, it seeks to expain the generation of those structures, either in terms of historica change or of deeper inguistic and experientia reaities. Reativism A phiosophica position that a points of view are equay vaid, and that a truth is reative to the individua. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xv

18

19 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 List of abbreviations HSDR NIHR Heath Services and Deivery Research Nationa Institute for Heath Research PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Anayses Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xvii

20

21 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Pain Engish summary Organisationa case studies typicay invove the cose, detaied examination of an organised body of peope with a particuar purpose, such as a business, government department or charity group. This research method can be used to understand activity and behaviour within a rea-ife context. However, the actua methods proposed for organisationa case studies are often not ceary described. We aimed to deveop a set of reporting standards for organisationa case study research reevant to the UK NHS. Reporting standards are a ist of criteria that can be used to improve the consistency, rigour and reporting of research, making it more accessibe and usefu to different audiences. We deveoped the standards by first coecting together practica information about reporting from pubished research methods texts and reevant organisationa case studies. We presented this information to an onine pane of experts as a ist of items. We reduced and refined these items over two rounds, unti ony those agreed to be essentia for the reporting of organisationa case studies remained. Utimatey, there was agreement among experts on 13 items, which together formed the basis of the reporting standards. Appication of these standards to existing organisationa case studies suggested that they can be used to improve the consistency, rigour and reporting of future research. We suggest that the reporting standards be further tested (and possiby refined) for use by researchers seeking pubic funding. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xix

22

23 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Scientific summary Background Case study is commony understood to be a method of research that engages in the cose, detaied examination of a singe exampe or phenomenon, and is an approach commony used to understand activity and behaviour within a rea-ife context. Organisationa case studies are concerned with an organised body of peope with a particuar purpose, such as a business, government department or charity group. When conducted we, organisationa case studies can provide insights into organisationa changes in heath care that are not easiy achieved through other study designs. They can be used to identify faciitators and barriers to the deivery of services and to hep understand the infuence of context; high-quaity organisationa case studies have been used to examine ways of working in acute care, primary care, menta heath services, residentia care and across the NHS more broady. Athough good-quaity studies wi be funded and pubished, some organisationa case study proposas submitted to the Nationa Institute for Heath Research (NIHR) s Heath Services and Deivery Research (HSDR) programme can be poory articuated and methodoogicay weak, raising the possibe need for pubication standards in this area. Reporting standards aready exist for a range of study designs, incuding randomised trias, observationa studies, systematic reviews, cinica case reports, quaitative research, reaist syntheses, meta-narrative reviews, diagnostic/prognostic studies, quaity improvement studies and economic evauations. However, a search of the EQUATOR (Enhancing the QUAity and Transparency Of heath Research) cearinghouse for reporting guideines suggests that to date no such standards have been reported for organisationa case studies. Objectives To deveop reporting standards for organisationa case study research, with particuar appication to the UK NHS. Methods The reporting standards were deveoped in three stages: 1. a rapid review of the existing iterature to identify content 2. a modified Dephi consensus process to deveop and refine content and structure 3. appication of the high-consensus Dephi items to two sampes of organisationa case studies to assess their feasibiity as reporting standards. Data sources Reevant case studies and methods texts were identified through searches of ibrary cataogues, key author searches, focused searching of heath and socia science databases and some targeted website searching. Participants Experts and parties interested in the conduct of organisationa case study research (methodoogists, research funders, journa editors, interested poicy-makers and practitioners) were approached to participate. Individuas were identified through the rapid review, persona contacts, and by contacting the foowing organisations: Heath Services Research Network, the Socia Research Association, the UK Evauation Society and the Nationa Centre for Research Methods. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xxi

24 SCIENTIFIC SUMMARY Review methods Items for the Dephi were identified from the foowing texts: organisationa case studies reating to an organised body of peope with a particuar purpose, such as a business, government department or charity group, identified from searches or from case study projects considered by HSDR as being of high quaity methodoogica texts providing practica advice specific to the conduct of organisationa case study research. Two reviewers independenty screened tites and abstracts, with disagreements resoved through discussion or consutation with a third reviewer. Data extraction was conducted by one reviewer and checked by a second. Extracted items were dedupicated and cassified into a provisiona framework: panning and study design data coection data anaysis reporting. Dephi consensus methods The provisiona framework and its constituent items were sent to the Dephi expert pane for rating. The Dephi consisted of two rounds: In the first round, participants were presented with a the unique items identified from the rapid review. They were asked to rate each item as being essentia, desirabe or not necessary for the reporting of organisationa case studies. Participants were aso asked whether or not the provisiona framework (grouping items into panning/design, data coection, anaysis and reporting) in which items were presented was appropriate, and were given the opportunity to adapt this aongside the content. In the second round, participants received a restructured ist of items incorporating feedback from the resuts of the first round. They were given the opportunity to identify the reporting items as being reevant to a, some or no organisationa case studies. In both rounds, the high-consensus threshod was set at 70% agreement among respondents for each item. The ist of items with high consensus after the second round was appied to previousy identified case study pubications in order to (1) determine the reevance of these items to the reporting of rea-word organisationa case studies and (2) better understand how the resuts of the Dephi consutation might best be impemented as a reporting standard. Resuts An initia poo of 103 unique reporting items was identified from 25 methodoogica texts; eight exampe case studies (17 pubications) and 12 exempar case studies, which had been provided by the HSDR programme as exampes of methodoogicay strong projects (16 pubications), did not provide any additiona unique items. Of 36 experts invited to take part in the Dephi consensus process, 19 (53%) responded to the first round invitation. Fifteen respondents competed the entire round 2 questionnaire, 14 of whom had aso taken part in the first round. The majority of respondents in round 1 were researchers (80%) with substantia experience of authoring or otherwise contributing to organisationa case study research. In the first round, 10 items met the predefined minimum 70% agreement eve for being essentia, with consensus ranging from 74% to 95%. xxii NIHR Journas Library

25 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 In the second round, a sighty greater proportion of respondents thought a reporting standard for reporting organisationa case studies was desirabe than did not, though severa were uncertain. Others suggested that the usefuness of any standards woud depend upon how and where they are appied. Respondents were simiary divided about whether or not a reporting standard woud be feasibe for organisationa case studies. Thirteen items were utimatey rated as shoud be reported for a organisationa case studies by at east 70% of respondents, with the degree of consensus ranging from 73% to 100%. As a whoe, exempar case studies considered methodoogicay strong by the HSDR programme more consistenty reported the high-consensus Dephi items than did case studies drawn from iterature more broady. Of eeven exempar pubications, six (55%) reported a 13 items, compared with just 3 out of 17 (18%) of the exampe organisationa case study pubications. The high-consensus items were transated into a set of 13 reporting standards grouped into four sections: describing the design describing the data coection describing the data anaysis interpreting the resuts. TABLE A Consensus standards for the reporting of organisationa case studies Reporting item Describing the design 1. Define the research as a case study 2. State the broad aims of the study 3. State the research question(s)/hypotheses 4. Identify the specific case(s) and justify the seection Describing the data coection 5. Describe how data were coected 6. Describe the sources of evidence used 7. Describe any ethica considerations and obtainment of reevant approvas, access and permissions Describing the data anaysis 8. Describe the anaysis methods Interpreting the resuts 9. Describe any inherent shortcomings in the design and anaysis and how these might have infuenced the findings 10. Consider the appropriateness of methods used for the question and subject matter and why it was that quaitative methods were appropriate 11. Discuss the data anaysis 12. Ensure that the assertions are sound, neither over- nor under-interpreting the data 13. State any caveats about the study Page number on which item was reported Page number of justification for not reporting Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xxiii

26 SCIENTIFIC SUMMARY Concusions These reporting standards aim to improve the consistency, rigour and reporting of organisationa case study research, thereby making it more accessibe and usefu to different audiences. These audiences incude research sponsors, who need to make decisions about whether to fund proposed case studies; ethics and research advisory groups, who require carity about the specific panned methods; peer reviewers, who need to be abe to evauate the robustness of a competed case study; and readers and poicy-makers, who need to understand how the findings of an organisationa case study might be interpreted and impemented. The reporting standards themseves are intended primariy as a too for authors of organisationa case studies. They briefy outine broad requirements for rigorous and consistent reporting, without constraining methodoogica freedom. Impemented propery, these shoud faciitate peer review of organisationa case studies and give greater confidence to the readers of this kind of research. Impications for research These reporting standards shoud be incuded as part of the submission requirements for a organisationa case studies seeking funding. Though these reporting standards do not mandate specific methods, if a reporting standard is not reported for egitimate methodoogica reasons, the onus is on the author to outine their rationae for the reader. Fina report manuscripts shoud be accompanied by a version of the reporting standards competed by the study author(s), and both documents shoud be made avaiabe to peer reviewers. Funding boards may want to coect feedback from users (incuding commissioners, authors, peer reviewers) in order to buid engagement with the concept of reporting standards for organisationa case studies and to coect evidence that coud be used to evauate and/or further refine these standards. Funding Funding for this study was provided by the HSDR programme of the NIHR. xxiv NIHR Journas Library

27 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Chapter 1 Background Case study is commony understood to be a method of research that engages in the cose, detaied examination of a singe exampe or phenomenon, and is an approach commony used to understand activity and behaviour within a rea-ife context. When conducted we, organisationa case studies can provide insights into organisationa changes in heath care that are not easiy achieved through other study designs. They can be used to identify faciitators and barriers to the deivery of services and to hep understand the infuence of context, and high-quaity organisationa case studies have been used to examine ways of working in acute care, 1 primary care, 2 menta heath services, 3 residentia care 4 and across the NHS more broady. 5 7 Yin 8 describes a case study to be the preferred research method when (1) the main research questions are how? or why? ; (2) a researcher has itte or no contro over behavioura events; and (3) the focus of the study is a contemporary (as opposed to historica) phenomenon. However, there is no set methodoogy for a case study and the term is often used oosey, but typicay combines quaitative and quantitative data coection with a strong observationa component. Case study research can be conducted from both reativist and positivist perspectives, and can be used to generate new theories, vaidate existing theories or address both of these matters. 9 An individua case can be studied aone to understand something about the case itsef and its contexts, or compared with other cases for iustrative, expanatory or evauative purposes. 10 The case study has been proposed as an appropriate method for describing, expaining, predicting or controing processes associated with phenomena at the individua, group or organisationa eve. 11 The majority of the Nationa Institute for Heath Research (NIHR) s Heath Services and Deivery Research (HSDR) programme-funded case studies are specificay concerned with description or expanation at the organisationa eve. In the past, many proposas for organisationa case studies submitted to the HSDR programme have been poory articuated and methodoogicay weak and were therefore unikey to deiver robust research findings. Specific areas of concern raised by the HSDR programme incuded: Absence of cear research questions that the case study method is intended to answer. Vagueness about samping frame/strategy. Proposas where it is not cear how organisations or sites were seected or what was the basis for samping. Insufficient theoretica basis. Many studies ack an organising theoretica framework; this can affect a stages, from samping of sites through to anaysis and how findings can add to the body of knowedge. Lack of carity about the unit of anaysis. Some weaker proposas wi not identify the unit of interest, whereas good case studies may incude data streams around the individua, team, organisation and wider system, and wi be expicit about the overa study design and interest. Lack of any cear pans for anaysis. Some proposas make no attempt to ook activey for data that chaenge emerging theories, findings or knowedge of systematic comparative case anayses. Many such studies are purey descriptive, without any expanatory power. Lack of carity about how data from a range of sources wi be integrated. Proposas increasingy caim to use reaist evauation methods for case study work, but make no attempt to estabish a programme theory, identify candidate mechanisms or describe other features of reaist evauation. Consequenty, the HSDR programme expressed an interest in identifying the characteristics of good-quaity case study research, and in devising quaity and pubication standards, with particuar appication to the NHS. More specificay, they described the need for a rapid evidence review aongside a Dephi or expert consensus-buiding exercise to identify eements of good practice and standards for reporting and pubication. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 1

28 BACKGROUND Athough some authors have proposed practica methodoogica guideines for case study research methods, 8,9 these have not been universay adopted. The broad diversity of approaches used within organisationa case studies and the contrasting paradigms that underpin these approaches mean that any attempt to deveop definitive methodoogica guidance in this area is ikey to be both highy contentious and resource-intensive. However, the abiity of research funders, peer reviewers and other research users to estabish methodoogica quaity is at east party contingent on the carity of expanation of the methods proposed or utiised. Indeed, severa of the concerns raised by the HSDR programme above specificay refer to vagueness or ack of carity around the reporting of proposed research methods. Reporting standards aready exist for a range of study designs, incuding randomised trias, 12 observationa studies, 13 systematic reviews, 14 cinica case reports, 15 quaitative research, 16 reaist syntheses, 17 meta-narrative reviews, 18 diagnostic/prognostic studies, 19 quaity improvement studies 20 and economic evauations. 21 However, to date, no such standards have been reported for organisationa case studies. By encouraging authors to consider how their methods are presented, the avaiabiity of an appropriate set of reporting standards for organisationa case studies aso has the potentia to improve research conduct in genera. A suitabe first step towards better conduct of organisationa case studies woud be to estabish agreement about what needs to be reported among the diverse group of researchers who undertake this kind of research. Shoud further guidance be needed about appropriate methods, the reporting standards can be used as a foundation on which to buid. The aim of this project has been to identify the characteristics of good-quaity organisationa case study research and devise reporting standards, with particuar appication to the NHS. Athough a range of opinions and experiences has been sought, the project has not been concerned with case studies outside the remit of the work funded by the HSDR programme. Therefore, it is not intended the reporting standards shoud be appied to case studies of individuas or to those conducted in other research fieds. In the first instance, we woud anticipate that these standards shoud be used to improve the standard of submissions to the HSDR programme. There may be further potentia for dissemination of the standards to the wider word of organisationa case study researchers. 2 NIHR Journas Library

29 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Chapter 2 Methods Research aim The aim of the project was to deveop reporting standards for organisationa case study research, with particuar appication to the UK Nationa Heath Service. Scope We deveoped the reporting standards in three stages, as shown in Figure 1: 1. a rapid review of the existing iterature to identify content for the standards 2. a Dephi consensus process to deveop and refine the fina set of standards 3. appication of the high-consensus Dephi items to two sampes of organisationa case studies to assess their feasibiity as reporting standards. Literature search Methods texts describing the purpose, design and conduct of organisationa case studies Exampe case studies reevant to the NHS Exempar case studies 1. Rapid review Item generation and provisiona framework 2. Consensus process Deveopment and refinement of framework and content Transating high-consensus Dephi items into reporting standards 3. Feasibiity assessment FIGURE 1 Outine of the research process. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 3

30 METHODS Rapid review A rapid review was used to generate items to popuate a provisiona framework for organisationa case studies. Systematic review methodoogy was used to identify artices and extract and synthesise data. Because of the rapid nature of the review, the process was ess exhaustive and contained ess detai than woud have been achievabe from a fu systematic review. Literature searching The aim of the search strategy was to identify materia about organisationa case study methods. It was anticipated that the iterature on this topic woud be found in textbooks, book chapters, journa artices and research methods guidance; therefore, the search strategy consisted of searches of ibrary cataogues, key author searches, focused searching of heath and socia science databases and some targeted website searching. Library cataogue searches The foowing ibrary cataogues speciaising in heath management iterature were searched to ocate books on case study methods: Heath Services Management Centre ONLINE (via the University of Birmingham; faciities/hsmc-ibrary/ibrary-resources/index.aspx) Heath Management Onine (via NHS Scotand; The King s Fund Library Database ( Key author searches Five authors, detaied in Box 1, featured prominenty in the initia iterature searches: David Byrne, Bent Fyvbjerg, Roger Gomm, Chares Ragin and Robert K Yin. Searches were carried out via Googe (Googe Inc., Mountain View, CA, USA) on each author to ocate any ists of their pubications. Pubication ists were found for each author on either their institution website or, where this was not avaiabe, through searches of Googe Books and Googe Schoar. Database searches Initia database test searches reveaed difficuties in ocating case study methodoogy iterature efficienty without retrieving arge numbers of irreevant resuts. Therefore, as this was a rapid review, a highy focused search strategy was deveoped on MEDLINE (via Ovid) to identify papers about organisationa case study methods. Focusing of subject headings, use of subheadings and searching in the tite-ony fied were utiised in the strategy. Searches were restricted to Engish-anguage papers. A more imited range of databases than usua for a fu systematic review was searched. In particuar, no specific databases of conference proceedings, theses or foreign-anguage studies were searched. Reevant databases covering iterature from heath, heath management and socia science were searched: MEDLINE & MEDLINE In-Process, Appied Socia Sciences Index and Abstracts (ASSIA), Heath Management Information Consortium, PsycINFO and the Socia Science Citation Index. The MEDLINE strategy was adapted for use in each database. Website searches The foowing websites were searched to identify any guidance documents on case study methods: ESRC Nationa Centre for Research Methods ( ESRC Research Methods Programme ( The Socia Research Association ( Methods@Manchester ( 4 NIHR Journas Library

31 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 BOX 1 Seected key pubications from authors featuring prominenty in searches David Byrne, Socia Scientist, Schoo of Appied Socia Sciences, University of Durham Byrne D. Compex reaist and configurationa approaches to cases: a radica synthesis. In Byrne D, Ragin C, editors. The SAGE Handbook of Case-based Methods. London: SAGE Pubications; pp Byrne D. Introduction: case-based methods: why we need them; what they are; how to do them. In Byrne D, Ragin C, editors. The SAGE Handbook of Case-based Methods. London: SAGE Pubications; pp Byrne DS. Compexity, configuration and cases. Theor Cut Soc 2005;22: Bent Fyvbjerg, Economic Geographer, Said Business Schoo, Oxford University Fyvbjerg B, Landman T, Schram S. Important next steps in phronetic socia science. In Fyvbjerg B, Landman T, Schram S, editors. Rea Socia Science: Appied Phronesis. Cambridge: Cambridge University Press; pp Fyvbjerg B, Landman T, Schram S. Rea Socia Science: Appied Phronesis. Cambridge: Cambridge University Press; Fyvbjerg B. Case study. In Denzin NK, Lincon YS, editors. The SAGE Handbook of Quaitative Research. 4th edn. Thousand Oaks, CA: SAGE Pubications; pp Roger Gomm, Socia Scientist, Schoo of Heath and Socia Wefare, The Open University Gomm R. Key Concepts in Socia Research Methods. London: Pagrave Macmian; Gomm R, Davies C. Using Evidence in Heath and Socia Care. London: SAGE Pubications; Gomm R, Hammersey M, Foster P. Case Study Method Key Issues, Key Texts. London: SAGE Pubications; Chares Ragin, Socioogist, Schoo if Socioogy, University of Caifornia at Irvine Ragin CC. Refections on casing and case-oriented research. In Byrne D, Ragin CC, editors. The SAGE Handbook of Case-based Methods. London: SAGE Pubications; pp Ragin CC, Schneider G. Case-oriented theory buiding and theory testing. In Wiiams M, Vogt P, editors. The SAGE Handbook of Methodoogica Innovations. London: SAGE Pubications; pp Ragin CC, Schneider G. Comparative poitica anaysis: six case-oriented strategies. In Amenta E, Nash K, Scott A, editors. The New Backwe Companion to Poitica Socioogy. Chichester: Backwe; pp Robert K Yin, socia scientist, COSMOS Corporation Yin RK. Appications of Case Study Research. 3rd edn. London: SAGE Pubications; Yin RK. Case study methods. In Cooper H, editor. APA Handbook of Research Methods in Psychoogy, Vo 2: Research Designs: Quantitative, Quaitative, Neuropsychoogica, and Bioogica. Washington, DC: American Psychoogica Association; pp Yin RK. Case Study Research: Design and Methods. Thousand Oaks, CA: SAGE Pubications; Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 5

32 METHODS Citation searching Citation searching on case study methods texts from key authors had been panned in the protoco. However, test citation searches identified arge numbers of resuts; therefore, given the rapid nature of the review, citation searching was not feasibe within the timescae. Records were managed within an EndNote ibrary (version X6; Thomson Reuters, CA, USA). After dedupication 3465 records in tota were identified. Further detais of the fu search strategies and resuts can be found in Appendix 1. Incusion and excusion criteria We sought to identify three sources of information: 1. methodoogica texts that reported on the methods used in conducting organisationa case study research 2. rea-word exampe case studies identified from the searches 3. methodoogicay sound exempar case studies identified by the NIHR HSDR programme. Incusion and excusion criteria are isted in the foowing sections. Methodoogica texts Texts were incuded if they: described the conduct of organisationa case studies, where organisationa means reating to an organised body of peope with a particuar purpose, such as a business, government department or charity group contained practica advice on conducting case study research. Texts were excuded if they were: concerned with case studies of individuas (e.g. describing a singe patient) concerned with quaitative, quantitative or mixed methods in genera, rather than case studies in particuar primariy conceptua or theoretica discussions without practica guidance. Methodoogica texts were not restricted by topic area. Thus reevant methodoogica texts from outside heath/socia services iterature, such as business and education, were eigibe for incusion. We focused on practica rather than conceptua texts to identify potentia items for reporting standards, but were mindfu that organisationa case studies can have different underying epistemoogica assumptions (e.g. positivist vs. reativist), and that some paradigms end themseves more easiy to practica advice than others. Exampe case studies These were incuded if they: reported an organisationa case study (as defined above) were undertaken in a UK NHS or socia services settings. The purpose of incuding the exampe case studies was to identify any additiona items for the Dephi consensus process (see Dephi consensus process) that had not aready been identified from the methods iterature. We therefore prioritised those organisationa case studies with particuar reevance to a UK NHS and socia services settings. 6 NIHR Journas Library

33 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Exempar case studies The funders of this review provided exampes of what they considered to be methodoogicay strong case study research projects funded by the NIHR HSDR programme. These were aso examined to identify further items to inform the Dephi consensus process. Seection of reevant evidence Methodoogica texts An initia examination of the EndNote ibrary identified a very arge number of irreevant records referring to research methods more broady, therefore we ran a search for case stud* in the tite or abstract in order to restrict the resuts to reevant methodoogica texts. Two reviewers (MR/ST) then independenty screened tites and abstracts, with disagreements resoved through discussion. Fu-text copies were obtained for potentiay reevant records and again screened independenty by the same two reviewers. Disagreements were resoved by discussion with a third reviewer (AE). EPPI-Reviewer version 4 (Evidence for Poicy and Practice Information and Co-ordinating Centre, Socia Science Research Unit, Institute of Education, University of London, UK) text-mining software was used by one reviewer (ST) to screen the remaining tites and abstracts to estabish whether or not any reevant texts coud have been missed by our restricted search. The text-mining process earned what were reevant texts as the reviewer progressed through screening and brought these to the top of the ist, enabing faster retrieva of fu texts for assessment and potentia incorporation into the review. A tites and abstracts were screened, with a decreasing number of texts being seected as the process continued. Fu-text copies were obtained for potentiay reevant records and screened for incusion by one reviewer (ST). A second reviewer (MR or AE) examined excuded records. Disagreements were resoved by discussion. Exampe case studies To identify exampe case studies we further restricted the resuts gathered from the EndNote ibrary by searching for the terms organisationa or organizationa in either the tite or abstract within the case stud* subset of resuts. One reviewer (ST) screened the tites and abstracts, obtained fu-text copies of potentiay reevant records and seected these for incusion. The seection was checked by a second reviewer (MR) and disagreements were resoved through discussion with a third reviewer (AE). Exempar case studies Tweve case study projects were identified as exempar case studies by the HSDR programme staff who commissioned the review. For each project, we downoaded the reevant commissioning brief and, where avaiabe, the protoco, fina report and journa artices from the HSDR website. Data extraction Data extraction forms were created to capture the key methodoogica components described in individua studies. The forms provided a standard framework whie accommodating different approaches; the authors own wording was used wherever possibe. The methodoogica texts were extracted first, beginning with the two most commony cited case study methods texts. 8,36 The remaining methodoogica texts were then extracted in reverse chronoogica order. For the subsequent data extraction, we tried to restrict extraction to additiona non-dupicate items; truy identica items identified from two or more sources (i.e. dupicates) were ony extracted once, though if two or more items were considered to be simiar (but non-identica), these were retained. Data from the exampe and exempar case studies were then extracted in a simiar way. For incuded case studies, we focused on identifying the reporting methods, rather than criticay appraising the underying Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 7

34 METHODS methodoogy; we aimed to deveop a generic reporting structure that coud be appied to a range of different types of organisationa case study. Data extraction was conducted by one reviewer (MR or ST) and checked by a second (MR, ST or AE). Disagreements were resoved through discussion. Synthesis To generate items for the Dephi consensus process, the individua data extraction forms from the methodoogica texts were combined into one overa document. A comprehensive and iterative process of refinement was then undertaken, combining and grouping simiar components and further removing dupicates. A simiar process was undertaken to add in any additiona components from the individua exampe and exempar case studies. An initia framework was created by broady grouping items by research stage, as foows: panning and study design data coection data anaysis reporting. This provisiona framework expanded and evoved as the items were extracted, synthesised and revised. Dephi consensus process The content of the framework was refined and deveoped through a modified Dephi consensus process. The Dephi technique is a structured and iterative method for coecting anonymous individua opinions from a pane with reevant expertise in the topic where consensus is required. 37 The basic principe is for the pane to receive successive questionnaires, each one containing the anonymous responses to the previous round, and for them to modify their responses unti consensus is reached. The Dephi consensus process was empoyed in order to obtain consensus from experts on the minimum set of reporting criteria that coud form the basis of standards for the reporting of future organisationa case studies. Design The reporting standards were deveoped over two rounds: In the first round, participants were presented with a the unique items identified from the rapid review. They were asked to rate each item as being essentia, desirabe or not necessary for the reporting of organisationa case studies. Participants were aso asked whether or not the provisiona framework in which items were presented was appropriate, and were given the opportunity to adapt this aongside the content. In the second round, participants received a restructured ist of items incorporating the resuts of the first round. Within each section, participants were first presented with high-consensus items (i.e. those receiving > 70% essentia responses in round 1), and given the opportunity to state whether such items shoud be reported by a organisationa case studies, specific types of organisationa case study or do not need to be reported. The remaining non-consensus items were ranked according to their positive/negative ratio of ratings from round 1. This ratio was cacuated for each item by dividing the sum of essentia and desirabe counts by not necessary counts. Consequenty, a ratio vaue of 1 woud indicate an even baance of positive and negative ratings. Participants were provided with each item and its corresponding ratio and again asked whether the item shoud be reported for a, some or no organisationa case studies. 8 NIHR Journas Library

35 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 In both rounds, the high-consensus threshod was set at 70% agreement among respondents for each item. This threshod was chosen as it refects a greater than 2 : 1 ratio of agreement to dissent, representing much stronger consensus than woud a simpe majority agreement threshod of 50% or greater. Each round was open for 3 weeks, with a reminder sent to non-responders at the end of the first week. Participants Experts and parties interested in the conduct of organisationa case study research (methodoogists, research funders, journa editors, interested poicymakers and practitioners) were invited to participate. Individuas were identified through the rapid review, persona contacts, and by contacting the foowing organisations: Heath Services Research Network; the Socia Research Association; the UK Evauation Society; and the Nationa Centre for Research Methods. A contacts were assured confidentiaity, with the aim of encouraging participation and openness, and a were invited to each round of the survey, incuding previous-round non-responders (uness they chose the option to withdraw from further contact). In order to assess representation of different stakehoder groups and identify any important differences in their responses, professiona characteristics were requested in each questionnaire. These incuded designation, topic area of interest, research method of interest and proportion of work reating to methodoogy. Instrumentation Questionnaires were administered eectronicay using on-ine survey software Quatrics (Quatrics, Provo, UT, USA) and a questionnaires were pioted before distribution. Anaysis A responses were coected in Quatrics for initia tabuation and anaysis. Subsequent anayses and outputs were produced in Microsoft Exce (Microsoft Corporation, 2010, Redmond, WA, USA). Where a respondent did not repy to a question, this vaue was recorded as missing. There was no imputation of missing vaues. Ethica approva Ethics approva for the consensus process was obtained from the University of York Heath Sciences Research Governance Committee. Invitees were promised anonymity and submission of competed questionnaires was taken as impied consent. Transating high-consensus Dephi items into reporting standards for organisationa case studies During the process of gathering the data from the rea-word exampe case studies and the exempar organisationa case studies provided by HSDR, we became interested in how these might match up to the reporting standards. Athough this step had not been part of the origina protoco, we decided to add an additiona step in the deveopment process. One reviewer (ST) appied the ist of high-consensus items, as far as was possibe in retrospect, to a identified exampe and exempar case study pubications. These were subjective decisions made by one reviewer and are not intended to be a criticism of the quaity of reporting in these pubications. Rather our aims were (1) to determine the reevance of these items to the reporting of rea-word organisationa case studies and (2) to better understand how the resuts of the Dephi consutation might best be impemented as reporting standards. The resuts of this appication are discussed in Transating high-consensus Dephi items into reporting standards for organisationa case studies. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 9

36

37 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Chapter 3 Resuts Rapid review The searches identified 3465 potentiay reevant references. After dedupication there were 2456 references, which were manuay screened, with 2348 excuded based on tite and abstract. Of the 1009 references screened with the assistance of text mining, 974 were excuded. Thirty-five records were identified from the screening of tites and abstracts during text mining, but no additiona texts were incuded after reading of fu texts. Foowing screening of fu texts we incuded: 25 methodoogica texts, 8 10,34 36,38 56 eight exampe case studies (17 pubications) and 12 exempar case studies (16 pubications) provided by the HSDR programme. 1 7,74 82 The study by Raine et a. 77,78 was described in pubications as a prospective observationa study, but contained many eements of a case study and was identified as an exempar of organisationa case study research. See Figure 2 for detais. Unique references retrieved from database searches (n = 3465) References screened with text mining assistance (n = 1009) References manuay screened (n = 2456) Excuded on tite and abstract (n = 974) Excuded on tite and abstract (n = 2348) Potentia reevant references to screen (n = 143) Methodoogica texts, n = 91 Exampe case studies, n = 52 References from 12 HSDR exempar organisationa case studies (n = 16) Fu texts excuded (n = 101) Unique references (n = 58) Methodoogica texts (n = 25) Exampe case studies (17 pubications) (n = 8) HSDR exempar case studies (16 pubications) (n = 12) FIGURE 2 A Preferred Reporting Items for Systematic Reviews and Meta-Anayses (PRISMA) 14 fow chart. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 11

38 RESULTS Rapid review: methodoogica texts Twenty-five methodoogica texts were incuded in the rapid review. 8 10,34 36,38 56 Dates of pubication covered 20 years, ranging from 1994 to One text 47 was received too ate to incude in the Dephi consensus process, but it incuded no new items to add to the fina ist. A number of key authors in the fied of case study methodoogy had been identified in the eary stages of our review (see Chapter 2, Rapid review, Literature searching, Key author searches). Other key authors were aso identified. A compete ist of these authors and their pubications is provided in Tabe 1. After reading the texts, we seected those that gave practica advice on conducting research. The most commony cited pubications were by the authors Yin 8,34,35,54 56 and Stake. 36 Therefore, the items for the Dephi consensus process were initiay drawn from six pubications by Yin 8,34,35,54 56 and three by Stake. 10,36,52 The remaining texts were read in reverse chronoogica order to identify any additiona items, with a decreasing number of new items found as we progressed back in time. See Appendix 2 for the compete ist of items together with authors. The anguage and paradigmatic assumptions reated to each extracted item are ikey to refect the position of the origina academic author. For exampe, the appication of concepts such as vaidity and reiabiity to case studies derives directy from the pubications of Yin. 8,34,35,54 56 Across a the incuded texts authors gave various definitions of case study research, made different paradigmatic assumptions, and recommended different methods. Rather than taking a particuar position, we aimed to capture a these variations for incusion in the first phase of the Dephi consensus process. 12 NIHR Journas Library

39 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 TABLE 1 List of methodoogica texts Author (year) Tite and pubication detais Crowe et a. (2011) 38 The case study approach. BMC Med Res Methodo 2011;11:100 Darke et a. (1998) 39 Successfuy competing case study research: combining rigour, reevance and pragmatism. Inf Syst J 1998;8: Fitzgerad and Dopson Comparative case study designs: their utiity and deveopment in organizationa research. (2009) 40 In Buchanan DA, Brynam A, editors. The SAGE Handbook of Organizationa Research Methods. London: SAGE Pubications; pp Gagnon (2010) 9 The Case Study as Research Method: A Practica Handbook. Québec City, QC: Presses de Université du Québec; 2010 Gibbert and Ruigrok The what and how of case study rigor: three strategies based on pubished work. Organ (2010) 41 Res Methods 2010;13: Gibbert et a. (2008) 42 What passes as a rigorous case study? Strategic Manage J 2008;29: Gigun (1994) 43 A case for case-studies in socia-work research. Soc Work 1994;39: Giham (2000) 44 Case Study Research Methods (Continuum Research Methods). London: Boomsbury Academic; 2000 Greene and David A research design for generaising from mutipe case studies. Eva Program Pann (1984) ;7:73 85 Hays (2004) 46 Hutchinson (1990) 47 Case study research. In demarrais K, Lapan SD, editors. Foundations for Research: Methods of Inquiry in Education and the Socia Sciences. Mahwah, NJ: Lawrence Erbaum Associates Pubishers; pp The case study approach. In Moody LE, editor. Advancing Nursing Science through Research (Vo. 2). Newbury Park, CA: SAGE Pubications; pp Huws and Dahmann Quaity Standards for Case Studies in the European Foundation. Dubin: European Foundation (2007) 48 for the Improvement of Living and Working Conditions; 2007 Kaarbo and Beasey A practica guide to the comparative case study method in poitica psychoogy. Poit Psycho (1999) ;20: Meyer (2001) 50 A case in case study methodoogy. Fied Methods 2001;13: Moore et a. (2012) 51 Stake (2005) 36 Case study research. In Lapan SD, Quartaroi MT, Riemer FJ, editors. Quaitative Research: An Introduction to Methods and Designs. San Francisco, CA: Jossey-Bass; pp Quaitative case studies. In Denzin NK, Lincon YS, editors. The SAGE Handbook of Quaitative Research. 3rd edn. Thousand Oaks, CA: SAGE Pubications; pp Stake (1995) 10 The Art of Case Study Research. Thousand Oaks, CA: SAGE Pubications; 1995 Stake (1994) 52 Thomas (2011) 53 Case studies. In Denzin NK, Lincon YS, editors. Handbook of Quaitative Research. Thousand Oaks, CA: SAGE Pubications; pp How to do your Case Study: A Guide for Students and Researchers. London: SAGE Pubications; 2011 Yin (2014) 8 Case Study Research: Design and Methods. Thousand Oaks, CA: SAGE Pubications; 2014 Yin (2012) 34 Appications of Case Study Research. 3rd edn. London: SAGE Pubications; 2012 Yin (2012) 35 Yin (2006) 56 Yin (1999) 55 Yin (1998) 54 Case study methods. In Cooper H, editor. APA Handbook of Research Methods in Psychoogy, Vo 2: Research Designs: Quantitative, Quaitative, Neuropsychoogica, and Bioogica. Washington, DC: American Psychoogica Association; pp Case study methods. In Green GL, Camii G, Emore PB, editors. Handbook of Compementary Methods in Education Research. Mahwah, NJ: Lawrence Erbaum Associates; pp Enhancing the quaity of case studies in heath services research. Heath Serv Res 1999;34: The abridged version of case study research. Design and method. In Bickman L, Rog DJ, editors. Handbook of Appied Socia Research Methods. Thousand Oaks, CA: SAGE Pubications; pp Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 13

40 RESULTS Rapid review: exampe case studies Eight exampe case studies (with 17 associated pubications) were incuded A the studies were conducted in Engand, with most reating to the NHS and one evauating prison menta heath in-reach services. 73 Dates of pubication ranged from 2004 to The methods, as reported by the authors, covered a variety of approaches (see Tabe 2 for detais). TABLE 2 Exampe case studies Author (year) Type Summary of case study Where conducted Author-reported case study methods Attree et a. (2008) 57,58 J The study expored patient safety in an Engish pre-registration degree nursing curricuum, based on the Nursing and Midwifery Counci 2002 curricuum guideines a Fied et a. R, J Phase 2: five in-depth organisationa case (2005) 59,65,68,69 studies were conducted with adut hospice bereavement support services in Engand Hutchinson and Purce (2010) 61 J Drawing on case study research in seven NHS trusts, the study considers the roe and management of ward managers and paramedic supervisors, focussing on their human resource management responsibiities Kyratsis et a. (2010) 62,63 R, J To understand the impact of differing organisationa capacity and contextua circumstances on technoogy seection, as we as the subsequent procurement and impementation of the technoogies in 12 Engish NHS trusts Nationa Nursing R, J Phase 2: issues of oca impementation of Research Unit (2009) 64 The Productive Ward programme in five NHS acute trusts NHS heath-care trusts, north of Engand NHS hospices, Engand NHS, south of Engand NHS trusts, Engand NHS acute hospitas, Engand Mutipe organisationa case studies In-depth mutipe organisationa case studies A mutipe case approach A quaitative, mutisite, comparative case study A mix of quaitative research methods The Offender Heath Research Network (2009) 70,73 R, J Evauating prison menta heath in-reach services using case study sites across the North West, North East and Yorkshire, South West, South East and London regions Prison and young offender institutions, Engand Quaitative anaysis and a mutipe case study approach a Payne et a. (2007) 65 J Case study research methods in end-of-ife care: refections on three studies a Payne et a. (2004) 60,65 67 R, J Phase 3: six in-depth organisationa case studies of community hospitas in the South East and South West of Engand to identify how paiative care for edery peope is deivered in practice from the perspectives of service users and service providers NHS hospices, Engand; NHS hospitas, Engand; NHS UK NHS community hospitas, Engand Refections on methods used in three case studies, incuding Payne et a. a In-depth mutipe organisationa case studies a Ros and Payne (2004) 65,72 J A mutipe case study design: the context and processes of chidhood bereavement services, the experiences of famiies who use them and the compexity of the contextua conditions that surround UK chidhood bereavement services NHS UK J, journa artice; R, report. a Linked organisationa case studies with methods further reported in Payne et a. 65 In-depth mutipe organisationa case study approach as part of a arger quaitative study 14 NIHR Journas Library

41 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Some of the case studies were parts of wider projects that incuded other methods of evauation. In such cases we focused ony on the methods used for the organisationa case studies. The eve of reporting of organisationa case study methods within individua pubications varied. After assessing a the pubications for each incuded organisationa case study, no new items were found to add to the Dephi consensus process. Exempar case studies Tweve case studies (16 pubications) funded by the NIHR HSDR programme were identified by the funder as being methodoogicay strong. 1 7,74 82 The methods, as reported by the authors, covered a variety of approaches (see Tabe 3 for detais). The ist of exempar case studies contains a number of competed and ongoing projects. Pubications incuded protocos, fina reports and journa artices. Most case studies were conducted in Engand, with one being conducted in a four countries of the UK. 80 A were conducted in the NHS. After a thorough reading of the pubications reating to case studies, no new items were identified to add to the Dephi consensus process. TABLE 3 Exempar case studies funded by the NIHR HSDR programme Chief investigator Type Project ID, tite and ink to HSDR project page Where conducted Author-reported case study methods Checkand K 2 R 08/1808/240: management practice in Primary Care Organisations: the roes and behaviours of midde managers and GPs Coss J 1 P 11/2000/05: the detection and management of pain in patients with dementia in acute care settings: deveopment of a decision too Drennan VM 74 R 09/1801/1066: investigating the contribution of physician assistants to primary care in Engand: a mixed-methods study Giard S 3 R 10/1008/15: new ways of working in menta heath services: a quaitative, comparative case study assessing and informing the emergence of new peer worker roes in menta heath services in Engand Primary care trusts Engand NHS Engand NHS Engand NHS Engand Quaitative case study methods Mutipe case studies with embedded units of anaysis Mixed methods (quaitative and quantitative) Comparative case study design continued Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 15

42 RESULTS TABLE 3 Exempar case studies funded by the NIHR HSDR programme (continued) Chief investigator Type Project ID, tite and ink to HSDR project page Where conducted Author-reported case study methods Goodman C 4 P 11/1021/02: optima NHS service deivery to care homes: a reaist evauation of the features and mechanisms that support effective working for the continuing care of oder peope in residentia settings Martin GP 5 7 R, J 09/1001/40: the medium-term sustainabiity of organisationa change in the Nationa Heath Service: a comparative case study of cinicay ed organisationa innovations McCourt C 75 R 10/1008/35: an ethnographic organisationa study of aongside midwifery units: a foow-on study from the Birthpace in Engand programme McDonad R 76 P 08/1809/250: evauation of the advancing quaity pay for performance programme in the NHS North West Raine R 77,78 R, J 09/2001/04: improving the effectiveness of mutidiscipinary team meetings for patients with chronic diseases: A prospective cohort study Rande R 79 P 12/5005/04: a reaist process evauation of robotic surgery: integration into routine practice and impacts on communication, coaboration and decision making Rycroft- Maone J 80 P 12/64/187: accessibiity and impementation in UK services of an effective depression reapse prevention programme: Mindfuness based cognitive therapy Waring J 81,82 R, J 10/1007/01: knowedge sharing across the boundaries between care processes, services and organisations: the contributions to safe hospita discharge and reduced emergency readmission J, journa artice; P, protoco; R, fina report. NHS Engand NHS Engand NHS Engand NHS Engand NHS Engand NHS Trusts, Engand NHS UK NHS Engand Reaist evauation Quaitative comparative case study Ethnographic study Mixed methods (quaitative and quantitative) Mixed methods (quaitative and quantitative) Reaist evauation In-depth case studies using exporatory and interpretive methods Ethnographic study 16 NIHR Journas Library

43 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Chapter 4 Dephi consensus process Items identified from the iterature After dedupication, a tota of 103 unique items were identified for incusion in the Dephi consensus process. See Appendix 2 for the fu ist of items. During the extraction process, the cassification of items evoved and expanded from four to the six foowing categories: 1. describing the design 2. background, context and theory 3. describing the data coection 4. describing the data anaysis 5. interpreting the resuts 6. sharing the resuts and concusions. These categories were used to structure the questionnaire, though respondents were given the opportunity to suggest additions or changes to this cassification (see Appendix 3). Round 1 resuts Response rate and participants Of 36 experts invited to take part in the Dephi consensus process, 19 (53%) responded to the first round invitation. A respondents competed the entire questionnaire. Foowing the distribution of questionnaires, the funder of this project was contacted by a earned society for socia science researchers, which expressed concerns about perceived assumptions underying the project. The three main concerns raised were (1) the difficuty in mandating standards of conduct for the wide variety of case study approaches, (2) the existence of a quaity contro system aready operating through peer review of the HSDR programme-funded project reports and (3) the risk of moving towards excessive standardisation. Four of the experts invited to participate in the Dephi consensus process co-signed the etter, three of whom aso went on to compete both rounds of the survey. The comments from these authors, as we as the concerns raised in the etter, were used to inform and refine the structure of the second round questionnaire, and are discussed further in Round 2 resuts. The characteristics of respondents to both rounds of the process are given in Tabe 4, and their research interests are described in Box 2. The majority of respondents in round 1 were researchers (80%), with substantia experience of authoring or otherwise contributing to organisationa case study research (see Tabe 4). Two respondents cassified themseves as research methodoogists, two others cassified themseves as having an editoria or reated pubishing roe, and one respondent was a research funder. Severa respondents expressed research interests reated to heath and/or socia care, and others an interest in different approaches to organisationa case study research (e.g. ethnography, quaitative case studies, comparative and theory-reated cases). Rating of items Respondents were asked to rate absoutey necessary items for reporting case studies as essentia, to rate usefu but non-essentia items as desirabe and rate any unnecessary, uncear, redundant, or meaningess items as not necessary. None of the 103 items was definitivey excuded by consensus (i.e. the proportion of not necessary ratings was beow 70% for every item). Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 17

44 DELPHI CONSENSUS PROCESS TABLE 4 Characteristics of respondents Number of respondents Characteristic Round 1 Round 2 Professiona roe Researcher 15 1 Research methodoogist 2 1 Journa editor/board member/pubishing 1 1 Other ( researcher and journa editor ) 1 1 How many organisationa case studies have you authored? > How many organisationa case studies have you been invoved with other than as an author? (e.g. peer review, commissioning, advisory roe) > What proportion of your work reates to research methodoogy? % % 4 2 > 60% NIHR Journas Library

45 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 BOX 2 Research interests of respondents (round 1 and round 2) Main area(s) of research interest reated to organisationa case studies Heath and socia care. Evauation of heath IT. Quaitative case studies. Deveopment of different modes of service deivery and interface between primary and secondary care. Organisation of care in hospita wards. Use of information technoogy in heath-care settings. Comparative cases; theory-reated cases. Evidence impementation; quaity improvement Pubic services broady heath, chidren s services, urban regeneration, disabiity services. Heath care. Change; impementation of evidence; maternity care; user and professiona experiences; ethnography. Heath services. Ethnography. Heath-care organisation. Research funder judging quaity of organisationa case study research. Aso editor for NIHR Journas Library reviewing quaity. Research into heath poicy. Hospitas. Quaity improvement and change management. Organisationa change. Reaist evauation, quaitative methods. Acute hospita settings. Reationships between organisationa structures and poicy outcomes. Tabe 5 shows the 10 items that met the predefined minimum 70% essentia consensus eve. The eve of consensus for these items ranged from 74% to 95%, with the highest consensus for describe how the data were coected and describe the sources of evidence used. None of the items cassified under the headings of background, context and theory or sharing the resuts and concusions met the 70% essentia consensus threshod. Among items faiing to meet the 70% essentia threshod, vaues ranged from 0% to 68%. In order to better inform respondents and to faciitate the rating process in round 2, these 93 non-consensus items were ranked according to their positive/negative ratio of ratings from round 1. This ratio was cacuated for each item by dividing the sum of essentia and desirabe counts by not necessary counts. Consequenty, a ratio of 1 woud indicate an even baance of positive and negative ratings. Where not necessary counts were 0, a vaue of 0.5 was used to aow the cacuation. Ratio vaues for the non-consensus subsequenty items ranged from 0.58 to 38. Round 1 comments A number of themes emerged from the comments given by respondents to round 1 (see Appendix 5 for a comments). Severa comments raised concerns about the phrasing of items. These fe into two categories: the inabiity to abe items because they were uncear, inappropriate or poory worded; and the impression that some items were overy focused on quantitative research and/or were informed by a rigid and predominanty positivist paradigm. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 19

46 DELPHI CONSENSUS PROCESS TABLE 5 Items identified as essentia by > 70% of respondents in round 1 Rating, n (%) Item Essentia Desirabe Not necessary Describing the design Define the research as a case study 14 (73.7) 5 (26.3) 0 State the broad aims of the study 16 (84.2) 3 (15.8) 0 State the research question(s)/hypotheses 15 (78.9) 3 (15.8) 1 (5.3) Data coection Describe how data were coected 18 (94.7) 1 (5.3) 0 Describe the sources of evidence used, for exampe documentation; archiva records; interviews; direct observations; participant-observation; physica artefacts Describe any ethica considerations and obtainment of reevant/approvas, access and permissions 18 (94.7) 1 (5.3) 0 15 (78.9) 4 (21.1) 0 Data anaysis Describe the anaysis methods 17 (89.5) 2 (10.5) 0 Interpretation Describe any inherent shortcomings in the design and anaysis and how these might have infuenced the findings Consider the appropriateness of methods used for the question and subject matter and why it was that quaitative methods were appropriate Discuss the data anaysis (was it conducted in a systematic way and was it successfu in incorporating a observations and deaing with variation?) 15 (78.9) 3 (15.8) 1 (5.3) 15 (78.9) 4 (21.1) 0 14 (73.7) 4 (21.1) 1 Severa other comments expicity noted that the appropriateness of certain items woud be context specific, and so a singe rating coud not be universay appied across different approaches. Other comments objected to the very notion of producing standards for the kind of contextuaised and creative interpretative processes that are often empoyed in quaitative research. None of the respondents suggested any changes to the six item categories. Round 2 resuts Deveopment of the questionnaire The responses from the first round were used to refine and deveop both the introductory information and the restructuring of the items in the next questionnaire, which was distributed in the second round of the Dephi consensus process. Introduction to round 2 The round 2 questionnaire was prefaced with an introduction that directy addressed the main concerns raised by respondents to the first round. 20 NIHR Journas Library

47 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 First, it was carified that whie the funders originay proposed a common quaity and reporting standard for organisationa case study research, the research team had anticipated that generic standards for the conduct of organisationa case studies woud not be possibe, and so chose from the start to focus on quaity of reporting rather than scientific quaity more broady (i.e. to identify any aspects of case study reporting that coud faciitate the reading and judgement processes used by peer reviewers and other audiences). However, in ight of the etter received by the society of socia science researchers and associated comments from round 1, respondents to round 2 were given the opportunity to expicity state whether they considered such reporting standards to be feasibe or desirabe. It was aso carified that the items presented in the Dephi consensus process were not created by the research team but were derived from the pubished academic iterature, using the authors own wording wherever possibe. Thus, the anguage used and the paradigmatic assumptions reated to each item ikey refect the position of the origina academic author. For exampe, the contentious appication of terms such as vaidity and reiabiity to case study research came directy from the pubished work of Yin. 8,34,35,54 56 The introduction to the exercise aso emphasised the research team s impartiaity regarding the fina content of the reporting standards, aong with the respondents prerogative to excude any items that they considered inappropriate, confusing, poory worded or meaningess. Item presentation Items were again grouped into six categories: (1) describing the design; (2) background, context and theory; (3) describing the data coection; (4) describing the data anaysis; (5) interpreting the resuts; and (6) sharing the resuts and concusions. Within each section, respondents were first asked to agree or disagree with the incusion of the high-consensus items from round 1 (> 70% essentia ) in generic reporting standards. They were then asked to either upgrade or discard the remaining ower-consensus items, which were presented in decreasing order of the positive-/negative-rating ratio. For a items, respondents had the opportunity to distinguish between items that shoud be reported for organisationa case studies in genera, those that shoud be reported for a particuar approach and those that did not need to be reported. Response rate and participants Fifteen respondents competed the entire round 2 questionnaire; 14 of these respondents (93%) had taken part in the first round and one respondent ony contributed to the second round. Athough a sighty greater proportion of respondents thought the estabishment of standards for reporting organisationa case studies was desirabe than did not, severa were uncertain (see Tabe 6 for response rates and a reated comments). Others suggested that the usefuness of any standards woud depend upon how they were appied (e.g. as a reference point for aspiration vs. a means of enforcing inappropriate standardisation) and where they are appied (e.g. heath service research vs. socioogy; impact on post-structuraist approaches). Respondents were simiary divided about whether or not meaningfu reporting standards woud be feasibe for organisationa case studies. Again, the issue of standards being possibe for some studies but not others was mentioned. As might be expected, given the very high proportion of overap between rounds, respondents had a simiar eve of case study experience and range of research interests as in round 1 (see Tabe 4). Appendix 6 contains a the free-text comments provided in round 2. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 21

48 DELPHI CONSENSUS PROCESS TABLE 6 Round 2 respondent opinions on the vaue and feasibiity of reporting standards for organisationa case studies Question No opinion Yes No Do not know Other Comments Did you take part in round 1 of this Dephi exercise? Do you think that a pubication standard for reporting organisationa case studies is desirabe? 93.3% 6.7% 0% 40% 26.7% 13.3% 20% It depends on the audience or community. Advanced ethnographic case studies targeted at anthropoogy, cutura studies, socioogy or poicy studies are arguaby distinct from Heath Services Research or tria research communities. Aso, how do post-structuraist or even narrative case accounts fit with the idea of standards? Standards might constrain creativity and imagination! A depends how it is used. It is one thing to have a standard that acts as a reference point or aspiration; it is another if this is used inappropriatey to enforce standards that are not universay suitabe for a research that might be subjected to it Yes but... recognise heterogeneity of case study research Do you think that a meaningfu pubication standard for reporting organisationa case studies is possibe? 6.7% 33.3% 26.7% 26.7% 6.7% For some types of studies and not others, I suspect Rating of items Items considered reevant to a organisationa case studies Thirteen items were rated as shoud be reported for a organisationa case studies by at east 70% of respondents. These incuded a 10 high-consensus items from the first round, pus three further items: identify the specific case(s) and justify the seection, ensure that the assertions are sound, neither over- nor under-interpreting the data and state any caveats about the study (Tabe 7). In round 2, the degree of consensus ranged from 73% to 100%, with four items ( state the broad aims of the study, describe how the data were coected, describe the sources of evidence used, and describe the anaysis methods ) achieving 100% consensus. For a 13 items, the degree of consensus was greater than in round 1. As in round 1, none of the items cassified under the headings of background, context and theory or sharing the resuts and concusions met the 70% consensus threshod. 22 NIHR Journas Library

49 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 TABLE 7 High-consensus items identified in round 2 Number of respondents (percentage of tota) Item Shoud be reported for a Shoud be reported for specific types Does not need to be reported Describing the design Define the research as a case study 13 (86.7) 0 2 (13.3) State the broad aims of the study 15 (100) 0 0 State the research question(s)/hypotheses 13 (86.7) 2 (13.3) a,b 0 Identify the specific case(s) and justify the seection [e.g. key case (good exampe, cassic or exempary case); outier case (showing something interesting because it is different from the norm); oca knowedge case (exampe chosen on the basis of persona experience or oca avaiabiity)] 11 (73.3) 1 (6.7) 3 (20) Describing the data coection Describe how data were coected 15 (100) 0 0 Describe the sources of evidence used (e.g. documentation; archiva records; interviews; direct observations; participant observation; physica artefacts) Describe any ethica considerations and obtainment of reevant approvas, access and permissions 15 (100) (86.7) 1 (6.7) c 1 (6.7) Describing the data anaysis Describe the anaysis methods 15 (100) 0 0 Interpreting the resuts Describe any inherent shortcomings in the design and anaysis and how these might have infuenced the findings Consider the appropriateness of methods used for the question and subject matter and why it was that quaitative methods were appropriate Discuss the data anaysis (was it conducted in a systematic way and was it successfu in incorporating a observations and deaing with variation) Ensure that the assertions are sound, neither over- nor under-interpreting the data 13 (86.7) 0 2 (13.3) 13 (86.7) 0 2 (13.3) 13 (86.7) 0 2 (13.3) 11 (73.3) 0 4 (26.7) State any caveats about the study 11 (73.3) 0 4 (26.7) a Many but not a. b I am wary of what this means for exporatory case studies where research questions are not fixed in advance. c NHS based ones. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 23

50 DELPHI CONSENSUS PROCESS Items considered unnecessary In the second round, 36 items (35%) were cassified as does not need to be reported by at east 70% of respondents (see Appendix 7). The degree of consensus ranged from 73% to 93%. This emphasises the much higher eve of consensus among respondents reative to that seen in the first round. Items considered reevant to specific types of case study Seventy-two items (70%) were considered by at east one respondent to be appropriate in certain contexts but not others. Methodoogica approaches identified by respondents incuded quantitative, quaitative, positivist, reaist evauation, expanatory case studies and participatory/action research. Other types of case study identified incuded NHS based, poicy-sponsored research and charity-funded evauations. Respondents very rarey expanded on these abes. However, there was no consensus that any item shoud be considered reevant to a particuar type of case study (where method-specific items were identified, agreement ranged from 0% to 33%). Items with no overa consensus Fifty-two items faied to meet the 70% consensus threshod for either incusion or rejection (see Appendix 8). Combining counts of shoud be reported for a organisationa case studies with counts of shoud be reported for the foowing type of organisationa case study... woud resut in just three additiona items achieving a 70% overa positive consensus [ state whether an inductive or deductive approach to the anaysis has been taken, discuss the samping (or case seection) and expanation of samping strategy and describe the data coection too(s) (e.g. questionnaire or observationa protoco, incuding a description of any pioting or fied testing of the too) (Tabe 8)]. TABLE 8 Items meeting 70% consensus ony when a and specific categories are combined Item Shoud be reported for a Shoud be reported for specific types Does not need to be reported Comments State whether an inductive (e.g. grounded) or deductive (hypothesis testing/theoretica framework) approach to the anaysis has been taken Discuss the samping (or case seection) and expanation of samping strategy Describe the data coection too(s) (e.g. questionnaire or observationa protoco, incuding a description of any pioting or fied testing of the too) 66.7% 6.7% 26.7% It shoud be obvious 66.7% 6.7% a 26.7% Studies of heterogeneous popuations of organisations a 60% 13.3% b,c 26.7% One that you want to pubish in a positivist journa b When new or idiosyncratic data coection methods were used c Comments indicated by a, b and c reate to the coumn shoud be reported for specific types. 24 NIHR Journas Library

51 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Chapter 5 Transating high-consensus Dephi items into reporting standards for organisationa case studies As an additiona step in the deveopment process, we appied the ist of high-consensus items to a exampe and exempar case study pubications identified earier in the review process, in order to (1) determine the reevance of these items to the reporting of rea-word organisationa case studies and (2) better understand how the resuts of the Dephi consensus process consutation might best be impemented as reporting standards. As stated in Chapter 2, Transating high-consensus Dephi items into reporting standards for organisationa case studies, these were subjective assessments appied in retrospect by one reviewer and were used as practica exampes to hep with the deveopment of the reporting standards and are not intended to be taken as a critica appraisa of the pubications. Exampe case studies The high-consensus reporting items were appied to a 17 pubications of the eight exampe organisationa case studies (Tabe 9). Five of the eight case studies pubished reports. 59,62,64,67,73 One report from the Nationa Nursing Research Unit appeared to be aimed at end users and contained itte methodoogica detai. 64 A the case studies had at east one journa pubication. One journa artice 65 provided some methodoogica detai for three of the incuded case studies and their pubications. 59,60,66 69,72 Two inked pubications exporing patient safety in the Engish pre-registration degree nursing curricuum met some of the criteria for describing the design, 57,58 though one did not state the research questions/ hypotheses (item 3) 58 and neither fuy identified the specific cases and justified seection (item 4). Both fuy described data coection (items 5 7), but ony partiay described the anaysis methods (item 8). Both poory reported items reated to interpreting the resuts (items 9 13). A simiar pattern of reporting was found in four inked pubications evauating adut hospice bereavement support services, which incuded a report, 59 two journa artices 68,69 and a stand-aone paper which refected on the methods empoyed in this and other case studies. 65 The journa artices make reference to the fu report, but the number of items reported was simiar across a the pubications. However, the journa artices do not state the research questions/hypotheses (item 3) and either did not report or ony partiay reported how the specific cases were identified and the seection was justified. Across these pubications, there was generay poor reporting on the items reating to interpreting the resuts (items 9 13). The overarching methodoogica paper by Payne et a. was pubished after the three case studies were pubished and so was not referenced in the report or journa artices. 65 A case study to examine managing ward managers for roes in human resource management generated ony one pubication; 61 this satisfied a the items for describing the design (items 1 4), coecting data (items 5 7) and describing the anaysis (item 8). However, items for interpreting the resuts were poory reported (items 9 13). A study to understand the impact of differing organisationa capacity and contextua circumstances on technoogy seection, procurement and impementation incuded a report 62 and a journa artice. 63 The items on describing the design (items 1 4) were a reported. Two questions reated to describing the data coection were reported (items 5 and 6), but the item reating to ethica considerations (item 7) was not reported for either pubication. The journa artice 63 reported a items for interpreting the anaysis, but the report did not. 62 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 25

52 TRANSLATING HIGH-CONSENSUS DELPHI ITEMS INTO REPORTING STANDARDS FOR ORGANISATIONAL CASE STUDIES TABLE 9 Assessing exampe case studies against the high-consensus Dephi items High-consensus items (see key) Category Case study Design Data coection DA Interpreting the resuts First author (pubication type) Attree 57 (J) Y Y Y N Y Y Y P P N P Y Y Cooke 58 (J) Y Y N P Y Y Y P N N P Y N 2 Fied 59 (R) Y Y Y Y Y Y Y Y P Y P Y N Reid 69 (J) Y Y N P Y Y Y Y N N P Y N Reid 68 (J) Y Y N N Y Y Y Y N N P Y N 3 Hutchinson 61 (J) Y Y Y Y Y Y Y P N N P Y N 4 Kyratsis 62 (R) Y Y Y Y Y Y N Y N N P Y Y Kyratsis 63 (J) Y Y Y Y Y Y N Y Y Y Y Y Y 5 Nationa Nursing Research Unit 64 (R) N Y P Y Y Y N/A P P N U U N Robert 71 (J) N Y N Y Y Y N Y N N Y Y Y 6 Offender Heath Research Network 73 (R) Y Y Y Y Y Y Y Y Y Y Y Y Y Ricketts 70 (J) Y Y N Y Y Y Y Y Y Y Y Y Y 7 Payne 65 (methods paper) (J) Y Y Y Y Y Y Y Y Y Y Y Y Y Payne 67 (R) Y Y Y Y Y Y Y Y Y Y Y Y Y Payne 66 (J) Y Y Y Y Y Y Y Y Y N Y Y Y Hawker 60 (J) Y N N Y Y Y Y Y P N P Y P 8 Ros 72 (J) Y Y N Y Y Y P Y N Y U U N DA, data anaysis; J, journa artice; N, no; N/A, not appicabe; P, partia; R, report; U, uncear; Y, yes. Items 1. Define the research as a case study. 2. State the broad aims of the study. 3. State the research question(s)/hypotheses. 4. Identify the specific case(s) and justify the seection. 5. Describe how data were coected. 6. Describe the sources of evidence used. 7. Describe any ethica considerations and obtainment of reevant approvas, access and permissions. 8. Describe the anaysis methods. 9. Describe the inherent shortcomings in the design and anaysis and how these might have infuenced the findings. 10. Describe the appropriateness of methods used for the question and subject matter and why it was that quaitative methods were appropriate. 11. Discuss the data anaysis (i.e. was it conducted in a systematic way and was it successfu in incorporating a observations and deaing with variation?). 12. Ensure that the assertions are sound, neither over- nor under-interpreting the data. 13. State any caveats about the study. 26 NIHR Journas Library

53 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 A study ooking at issues of oca impementation of The Productive Ward programme incuded a report and a journa artice. 64,71 The report appeared to be aimed at end users. Three items were reported or partiay reported for design (items 2 to 4), two for data coection (items 5 and 6), one partiay reported for data anaysis (item 6), and one partiay reported for interpretation of resuts (item 9). 64 The journa artice reported more of the high-consensus Dephi items. Two items were reported in design (items 2 4), two in data coection (items 5 and 6), one in data anaysis (item 9) and three for interpreting the resuts (items 11 13). 71 One report 73 of a case study evauating prison menta heath in-reach services reported a high-consensus Dephi items. The associated journa artice 70 reported 12 of the 13 criteria, but did not report research questions/hypotheses (item 3). A case study used to identify how paiative care for edery peope is deivered was pubished as a report 67 and two journa artices, 60,66 as we as being inked to the methods paper mentioned earier. 65 The report reported a high-consensus Dephi items. 67 Items describing the design (items 1 4) were we covered in one artice, 66 but the other artice 60 reported neither the broad aims of the study (item 2) nor the research questions/hypotheses (item 3). Reporting of the data anaysis and interpreting the resuts was generay we covered in one artice 66 but ess so in the other pubication. 60 Both the report and journa artices were pubished before the Payne et a. methodoogica paper 65 and therefore did not expicity reference it. A case study examining the context and processes of chidhood bereavement services 72 was aso inked to the above-mentioned methods paper. 65 This journa artice did not state the research questions or hypotheses (item 3) and ony partiay reported on ethica considerations (item 7). Items reating to interpreting the data were aso poory reported. The journa artice was pubished before the Payne et a. methodoogica paper 65 and therefore did not expicity reference it. In summary, two pubications reported ess than 50% of the items, 58,64 eight reported between 50% and 70% of items, 57,60 62,68,69,71,72 four reported over 70% of items, 59,63,70,66 and three reported a the high-consensus items. 65,67,73 Across a pubications, the items describing the design (items 1 4), data coection (items 5 7) and the anaysis (item 8) were argey reported. There was variation in reporting on interpretation of resuts and severa studies either did not report or ony partiay reported their methods. There was no cear pattern in the number of items being reported between journa artices and reports. This was a reativey sma sampe of pubications aimed at different audiences, so it woud not be appropriate to draw concusions on eves of reporting within different types of pubications. These pubications covered a range of case study methodoogy and were aimed at different audiences (e.g. end users); therefore, the ack of reporting shoud not be taken to mean a ack of quaity in the methods used, nor as impied criticism of the origina authors. Exempar case studies Of the 12 exempars, ony seven had pubished reports. 2,3,7,74,75,77,81 Of these, two had a singe additiona journa artice and one had two reated journa artices. 5,6,78,81 The 13 high-consensus Dephi items of reporting standards were appied to each of the 11 pubications (Tabe 10). Six out of 11 of the pubications reported a 13 of the high-consensus Dephi items. 2,3,74,75,81,82 Three pubications for one case study did not fuy meet a the items. 5 7 One report, which incuded data on foow-up to previous case study sites, did not expicity state that the authors had ethica approvas, access or permissions (item 7). 7 The authors appeared to partiay describe shortcomings in the design and anaysis and how these might have infuenced the findings (item 9). They ony partiay considered the appropriateness of methods used (item 10) and did not state any caveats about the study (item 13). Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 27

54 TRANSLATING HIGH-CONSENSUS DELPHI ITEMS INTO REPORTING STANDARDS FOR ORGANISATIONAL CASE STUDIES TABLE 10 Assessing the HSDR programme exempar case studies against the high-consensus Dephi items High-consensus items (see key) Category Case Study Design Data coection DA Interpreting the resuts First author (pubication type) Checkand 2 (R) Y Y Y Y Y Y Y Y Y Y Y Y Y 2 Drennan 74 (R) Y Y Y Y Y Y Y Y Y Y Y Y Y 3 Giard 3 (R) Y Y Y Y Y Y Y Y Y Y Y Y Y 4 Martin 7 (R) Y Y Y Y Y Y N Y P P Y Y N Martin 6 (J) Y Y N Y Y Y Y Y N N Y Y N 5 Currie 5 (J) Y Y N Y Y Y N Y P P Y Y P 6 McCourt 75 (R) Y Y Y Y Y Y Y Y Y Y Y Y Y 7 Raine 77 (R) N Y Y Y Y Y Y Y Y Y Y Y Y Raine 78 (J) N Y Y Y Y Y Y Y Y Y Y Y Y 8 Waring 82 (R) Y Y Y Y Y Y Y Y Y Y Y Y Y Waring 81 (J) Y Y Y Y Y Y Y Y Y Y Y Y Y DA, data anaysis; J, journa artice; N, no; N/A, not appicabe; P, partia; R, report; U, uncear; Y, yes. Items 1. Define the research as a case study. 2. State the broad aims of the study. 3. State the research question(s)/hypotheses. 4. Identify the specific case(s) and justify the seection. 5. Describe how data were coected. 6. Describe the sources of evidence used. 7. Describe any ethica considerations and obtainment of reevant approvas, access and permissions. 8. Describe the anaysis methods. 9. Describe the inherent shortcomings in the design and anaysis and how these might have infuenced the findings. 10. Describe the appropriateness of methods used for the question and subject matter and why it was that quaitative methods were appropriate. 11. Discuss the data anaysis (i.e. was it conducted in a systematic way and was it successfu in incorporating a observations and deaing with variation?). 12. Ensure that the assertions are sound, neither over- nor under-interpreting the data. 13. State any caveats about the study. The two journa artices associated with the report aso did not meet a the items. 5,6 One artice, which was party inked with the main HSDR programme-funded project detaied in the report, did not state the research question/hypotheses (item 3), nor did it describe any shortcomings in the design (item 9), consider the appropriateness of methods used (item 10) or state any caveats about the study (item 13). 6 The other artice, pubished prior to the report, aso did not state the research question/hypotheses (item 3). 5 There was no reporting of ethica considerations (item 7). There was ony partia reporting of any inherent shortcomings in the design and anaysis and how these might infuence findings (item 9), consideration of the appropriateness of methods used (item 10) and caveats about the study (item 13). A report 77 and a inked journa artice 78 reported 12 out of 13 of items, but did not define the research as a case study (item 1). The authors of this study stated it was a prospective observationa study but, because it contained many eements of an organisationa case study, it was recommended by the HSDR programme for this project. 28 NIHR Journas Library

55 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 In summary, one exempar pubication 5 reported 61% of the reporting standards, two 6,7 reported 69% and six 2,3,74,75,81,82 reported a of the reporting standards. The journa artices argey reported the same criteria as the corresponding reports. As with the exampe case studies above, these pubications covered a range of case study methodoogy and were aimed at different audiences; therefore, the ack of reporting shoud not be taken as an indication of a ack of quaity in the methods used in the studies themseves, nor as impied criticism of the origina authors. As a whoe, the exempar case studies (which had been considered methodoogicay strong by the HSDR programme team) more consistenty reported the high-consensus Dephi items than did the exampe case studies drawn from the review searches. Of 11 exempar pubications, six (55%) reported a 13 items, compared with just 3 out of 17 (18%) of the exampe organisationa case study pubications. Generic consensus-based reporting standards for organisationa case studies The exempar organisationa case studies identified by the HSDR programme as being of high quaity were far more consistent with the high-consensus Dephi items than were a group of exampe case studies identified purey on the basis of topic reevance. If the atter group of studies are representative of the wider fied of organisationa case study research, then there is ceary scope to use the identified items to improve the improve the consistency and rigour of reporting in this area. Though the high-quaity case studies used different methodoogica approaches, they were consistent with one another on the high-consensus Dephi reporting items. This suggests that, athough these items can detect consistency and rigour of reporting, they are aso sufficienty generic to be appied to a variety of organisationa case study methods. The fact that journa artices sometimes satisfied more items than onger reports for the same case study suggests that the ength of a pubication is not necessariy reated to how ceary the research methods are reported. This may be a deiberate choice. For exampe, authors may choose to excude certain items from a report aimed at practitioners or poicy-makers, yet incude those same items in an academic journa artice aimed at other researchers. Simiary, there may be egitimate methodoogica reasons for a particuar item not being reported. For exampe, a researcher conducting a purey exporatory case study might not consider it appropriate to state an initia research question or hypothesis (item 3 on the reporting standards); in this case it woud be perfecty egitimate to briefy outine the justification for not doing so in the report. However, it is not aways obvious whether the absence of certain information is deiberate or an oversight; any reporting standards for organisationa case studies shoud be aware of this distinction. Therefore, unike reporting standards such as PRISMA, which mandate the incusion of every item in a report, 14 the reporting standards proposed in Tabe 11 require the author to refer to a pace where the reporting item was reported or where justification for the absence of the item can be found. This approach intends to baance the research freedoms of the knowedgeabe researcher with the information needs of the end user. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 29

56 TRANSLATING HIGH-CONSENSUS DELPHI ITEMS INTO REPORTING STANDARDS FOR ORGANISATIONAL CASE STUDIES TABLE 11 Consensus standards for the reporting of organisationa case studies Reporting item Describing the design 1. Define the research as a case study 2. State the broad aims of the study 3. State the research question(s)/hypotheses 4. Identify the specific case(s) and justify the seection Describing the data coection 5. Describe how data were coected 6. Describe the sources of evidence used 7. Describe any ethica considerations and obtainment of reevant approvas, access and permissions Describing the data anaysis 8. Describe the anaysis methods Interpreting the resuts 9. Describe any inherent shortcomings in the design and anaysis and how these might have infuenced the findings 10. Consider the appropriateness of methods used for the question and subject matter and why it was that quaitative methods were appropriate 11. Discuss the data anaysis 12. Ensure that the assertions are sound, neither over- nor under-interpreting the data 13. State any caveats about the study Page number on which item was reported Page number of justification for not reporting 30 NIHR Journas Library

57 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Chapter 6 Discussion The fina consensus-based reporting standards consist of 13 unique items grouped into four sections (see Tabe 11): 1. describing the design 2. describing the data coection 3. describing the data anaysis 4. interpreting the resuts. These standards aim to improve the consistency, rigour and reporting of organisationa case study research, thereby making it more accessibe and usefu to different audiences. These audiences incude research sponsors who need to make decisions about whether or not to fund proposed case studies; ethics and research advisory groups who require carity about the specific panned methods; peer-reviewers who need to be abe to evauate the robustness of a competed case study; and readers and poicy-makers who need to understand how the findings of an organisationa case study might be interpreted and impemented. Though severa items in the reporting standards refer to the conduct of case study research, the standards are not intended to be a guide on how to undertake an organisationa case study. There are mutipe texts that address methodoogy in this area at great ength, many of which informed the initia stage of this project. 8 10,34 36,38 56 Any checkist mandating specific case study methods woud be far more engthy than the proposed reporting standards, woud be difficut to impement universay across different research contexts and paradigms and woud ikey encounter resistance from some sections of the research community. Nevertheess, it can be seen that the majority of specific concerns raised by the HSDR programme when commissioning this work (e.g. absence of cear research questions, ack of carity about how cases were seected, ack of cear anaysis pans, absence of information about data sources) are directy addressed in the fina set of reporting standards. Perhaps surprisingy, none of the items that might have addressed a concern about case studies having an insufficient theoretica basis reached sufficienty high consensus to be incuded in the fina set of standards. We might have expected a greater proportion of respondents to consider the theoretica or methodoogica underpinnings of the research as vita to understanding how researchers interpret their resuts. But in fact, none of the items cassified as background, context and theory met the high-consensus threshod. It coud be that (a) the group of experts consuted truy did not consider the items presented in this category to be sufficienty important, (b) the experts beieved that this concern woud be addressed by another item (e.g. ensure that the assertions are sound, neither over- nor under-interpreting the data ) or (c) this is an artefact of the Dephi process. As there was high consensus among experts that the 13 items incuded in the reporting standards shoud be reported for a organisationa case studies, these items tend towards the generic rather than the specific. In fact, rather than being excusive to organisationa case studies, severa of the items refect good practice for the reporting of research more generay, and are simiar to items on reporting standards for other research designs This raised a concern that the reporting standards may be so generic as to be of itte vaue for the intended aim of improving the consistency and rigour of reporting of organisationa case study research. However, appying the standards to both exempar and exampe case studies suggested that (a) there are pubished organisationa case studies that do not meet these standards, (b) such studies coud be reported in a manner that meets the standards without impacting on their underying methodoogy and (c) exempar organisationa case studies identified by HSDR as being of high quaity are generay consistent with these standards, whereas the variation in standards is much greater among the exampe organisationa case studies that we reviewed (see Chapter 5). Consequenty, we beieve that even this short ist of reativey generic reporting items does have the potentia to improve the standard of reporting among organisationa case studies conducted in the NHS. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 31

58 DISCUSSION Severa experts expressed concerns about the risk of using a checkist, particuary if this were used in an attempt to standardise organisationa case study research methods. Given the generic nature of many of the items incuded in the reporting standards, we beieve that the risk of it being used to constrain research methodoogy is minima. In addition, the fina reporting standards are structured in such a way as to emphasise their primary intention as a means of supporting authors to report their research proposas and manuscripts, rather than as a too for standardisation of methods. Consequenty, for each item, the reporting standards provide the research author with the opportunity to acknowedge where the item has been reported, or to expain why the item has (egitimatey) not been reported. In either case, the authors wi have met the requirements of the reporting standards by providing the end user with important information about the design and purpose of the study. The standards are not prescriptive: authors are aowed to exercise judgement about how much information they choose to provide. As seen in Chapter 5, these reporting standards can be met in even reativey brief pubications. Severa reporting items incuded in the Dephi consensus process were considered essentia by a majority of respondents but, faiing to meet the pre-defined consensus threshod, they have not been incuded in the standards presented here. However, there may be an opportunity to expand and/or refine the current reporting standards further, possiby after it has been appied by authors of newy conducted organisationa case studies. Strengths of this project Unike a traditiona Dephi consensus process, in which items are generated by respondents and then refined in subsequent rounds by the same respondents, we expedited the process by deriving an initia poo of items from a rapid review of the methodoogica iterature reevant to organisationa case studies. These items were then rated in two rounds by a Dephi pane of experts, a of whom had direct invovement with case study research. This approach aimed to ensure that both the generation and refinement stages were informed by expert knowedge within the short time frame avaiabe for the project. The research team made concerted efforts to avoid infuencing the content or outputs of the review and consutation processes, and the processes themseves have been reported as ceary as possibe to maximise transparency and avoid bias. Aongside the review of methodoogica iterature, we aso examined two groups of case studies: high-quaity exempar studies identified as such by the NIHR s HSDR programme and a group of topic-reevant exampe studies obtained from the wider iterature. These ensured that the project was informed by rea-word research practices and aso provided an opportunity to check the vaidity of high-consensus Dephi items for incusion in the reporting standards. Evidence showing a discrepancy between exempar and exampe case studies in terms of performance against these items impies that there is scope for greater rigour and consistency of reporting in this area. Whie the best organisationa case studies (such as the exempars) woud not be much improved by these reporting standards, many other studies ceary woud be improved. Some Dephi respondents expressed concerns that the proposed reporting standards might constrain methodoogica freedom, particuary for researchers using quaitative or interpretive methods. Utimatey, ony the most generic items met the minimum consensus threshod. Some of the items are not even specific to the reporting of case studies, but to good reporting practice for empirica research more broady. Consequenty, we beieve that the reporting standards are sufficienty broad to encompass the various different approaches and paradigms that fa under the umbrea of organisationa case study research. Evidence from the exempars suggested that a we-reported case study is ikey to be consistent with the proposed reporting standards, regardess of the specific research methods or underying epistemoogica paradigm. Shoud an item from the reporting standards be truy inappropriate to the specific case study method, authors are given the opportunity to make this cear (with appropriate 32 NIHR Journas Library

59 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 methodoogica justification). This approach aims to baance the needs of readers concerned about methodoogica quaity with the methodoogica freedom of authors by pacing a strong emphasis on transparency. Consensus among the items incuded in the reporting standards was generay high, with 100% of respondents in round 2 agreeing that severa items shoud be reported for every organisationa case study. If the Dephi respondents are representative of knowedgeabe case study researchers more broady, we woud expect these standards to be acceptabe to the wider research community. Weaknesses of this project As stated earier, the research team attempted wherever possibe to avoid introducing bias or persona preferences into the review and consutation processes. For exampe, we intentionay avoided excuding or substantiay rewording items identified in the methods iterature wherever possibe, eaving any decisions about the vaue of these items to the Dephi pane. However, we were aware that this meant that some of the items coud have been worded more ceary or precisey. Athough we were keen not to risk changing the origina authors meaning during the research process, any future pioting of the reporting standards coud provide an opportunity to refine the exact wording of the incuded items. The tota number of respondents was reativey sma, though this is frequenty the case in Dephi research studies. As the primary aim is to identify the eve of consensus among experts, rather than generaising to a arger popuation, obtaining a sufficient degree of expertise and representative pane can be considered more important than obtaining a arge sampe. Data coected in this Dephi suggested that respondents had an appropriate eve of expertise and hed a range of views regarding the deveopment of reporting standards, from the enthusiastic to the sceptica. A minority of Dephi respondents suggested that the materia presented in the Dephi had a strongy positivist focus. This impression may have been a consequence of the data extraction process, in which discrete practica items reated to reporting were extracted from the methods iterature. This meant that authors ike Yin (who writes in a predominanty didactic pragmatic stye) were more strongy represented among the initia poo of items than were authors that focused on higher-eve abstractions or theoretica issues (e.g. Hammersey, Gomm, Fyvbjerg). The perception of positivist bias reated mainy to items derived from Yin, but it shoud be noted that these items typicay faied to meet the required consensus threshod, and so were not incuded in the fina reporting standards. Athough the reporting standards might not refect the terminoogy used in fieds such as anthropoogy or geography, they are ikey to be appropriate to the types of organisationa case study most frequenty funded by the HSDR programme. An aternative approach to the identification of items might have been to incude an earier item generation round in the Dephi consensus process, but time and resource constraints prevented this from being possibe. Some reporting items considered necessary by a strict majority of respondents (i.e. > 50% agreement) were not incuded in the fina reporting standards. However, a minimum 70% threshod was intentionay chosen as this gives a greater than 2 : 1 ratio of agreement to dissent, which more accuratey refects high consensus among the Dephi pane, particuary with a sampe size such that an individua respondent score constitutes amost 7% of the tota. Utimatey, most of items in the reporting standards far exceeded the 70% threshod. Just three of the subthreshod items achieved an overa positive consensus by combining report for specific case studies ratings with report for a case studies ratings ( state whether an inductive or deductive approach to the anaysis has been taken, discuss the samping (or case seection) and expanation of samping strategy and describe the data coection too(s) ). If the reporting standards were to be expanded beyond the 13 current items, based on the avaiabe data these woud be the most ikey candidates for addition. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 33

60 DISCUSSION There is aways a risk that easiy appied checkists can be impropery used as a substitute for proper methodoogica understanding by ess experienced researchers. The reporting standards presented here are intended to improve the transparency of reporting of organisationa case studies and have been presented in a format to ensure that they are used for this purpose. Athough awareness of these standards may improve the conduct of organisationa case studies, they currenty outine the minimum requirements for reporting and shoud not be considered a simpe checkist for estabishing methodoogica quaity. Meeting a 13 standards wi not guarantee that an organisationa case study has been we conducted, but it shoud provide readers with a sufficient understanding of how the case study was undertaken. 34 NIHR Journas Library

61 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Chapter 7 Concusions Key concusions from the project The reporting standards presented here are intended primariy as a too for authors of organisationa case studies. They briefy outine broad requirements for rigorous and consistent reporting, without constraining methodoogica freedom. They are not intended to be used as a critica appraisa too but, by improving reporting quaity, hopefuy they wi prove usefu for research funders, peer reviewers, journa editors and readers. Currenty, it appears that not a organisationa case studies report the items isted in the reporting standards but, if impemented propery in future, these standards shoud faciitate peer review of organisationa case studies and give greater confidence to readers. In genera, the proposed standards simpy require authors to acknowedge key stages of the research process. Appying the standards is unikey to be onerous, nor to resut in a change of methods or a great dea of extended detai in study reports. Impications for research Athough the proposed reporting standards are based on a high eve of consensus and have face vaidity, their true vaue cannot be fuy estabished unti they have been appied in practice. In the first instance, we propose that these reporting standards be incuded as part of the submission requirements for a organisationa case studies seeking pubic funding. Athough the fu set of standards can ony be met once the study has resuts to report, it might aso be usefu to make items 1 to 8 of the standards avaiabe to authors at the proposa stage. It coud be emphasised that cear reporting wi be of benefit to reviewers and readers (and utimatey to the authors themseves). Though these reporting standards do not mandate specific methods, if a reporting item is not reported for egitimate methodoogica reasons, the onus is on the author to outine their rationae for the reader. Fina report manuscripts shoud be accompanied by a version of the reporting standard pro forma competed by the study author(s), and both documents shoud be made avaiabe to peer reviewers. Funding boards may want to coect feedback from users (incuding commissioners, authors and peer reviewers) about the impementation of these standards. They may be more straightforward to impement for some forms of organisationa case studies than others (e.g. stand-aone organisationa case studies vs. case studies embedded within a arger study design). As mentioned previousy, we anticipate that some authors may fee that reporting standards are not reevant or necessary for organisationa case studies and that others may consider the standards proposed here to be too generic. However, consutation with research authors wi be necessary to buid engagement with the concept of reporting standards for organisationa case studies among various audiences, and to coect evidence that coud be used to evauate and/or further refine the existing standards. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 35

62 CONCLUSIONS Impications for practice We wi submit the proposed reporting standards to the EQUATOR (Enhancing the QUAity and Transparency Of heath Research) network for consideration. This is an internationa initiative that seeks to improve the reiabiity and vaue of pubished heath research iterature by promoting transparent and accurate reporting and wider use of robust reporting guideines. Its ibrary for heath research reporting ( contains 247 reporting guideines, of which 60 reate to observationa research methods. Currenty, none of these are expicity concerned with the reporting or organisationa case studies. 36 NIHR Journas Library

63 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Acknowedgements Contributions of authors Mark Rodgers and Sian Thomas carried out study seection, data extraction, critica appraisa and write up of the report. A searching was conducted by Meissa Harden, who aso wrote the search sections of the report and commented on the draft report. Giian Parker and Andrew Street provided expertise and advice, contributed to the deveopment of the protoco and commented on drafts of the report. Aison Eastwood carried out study seection, data extraction, critica appraisa and write up of the report. Advice James Thomas, of the University Coege London Institute of Education, provided expertise, advice and assistance with text mining. The review authors are gratefu to the Dephi pane of experts whose contribution made this work possibe. Data archiving A avaiabe data can be obtained from the corresponding author. A data wi be shared in a way that safeguards the confidentiaity and anonymity of respondents. Data sharing statement A avaiabe data can be obtained from the corresponding author. A data wi be shared in a way that safeguards the confidentiaity and anonymity of respondents. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 37

64

65 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 References 1. Dowding D. The Detection and Management of Pain in Patients with Dementia in Acute Care Settings (Protoco). Southampton: NIHR Heath Services and Deivery Research programme; Checkand K, Snow S, McDermott I, Harrison S, Coeman A. Management Practice in Primary Care Organisations: The Roes and Behaviours of Midde Managers and GPs. Fina report. Southampton: NIHR Service Deivery and Organisation programme; Giard S, Edwards C, Gibson S, Hoey J, Owen K. New ways of working in menta heath services: a quaitative, comparative case study assessing and informing the emergence of new peer worker roes in menta heath services in Engand. Heath Serv Deiv Res 2014;2(19). 4. Goodman C. Optima NHS Service Deivery to Care Homes: A Reaist Evauation of the Features or Mechanisms that Support Effective Working for the Continuing Care of Oder Peope in Residentia Settings. Southampton: NIHR Heath Services and Deivery Research; Currie G, Lockett A, Finn R, Martin G, Waring J. Institutiona work to maintain professiona power: recreating the mode of medica professionaism. Organ Stud 2012;33: / Martin GP, Weaver S, Currie G, Finn R, McDonad R. Innovation sustainabiity in chaenging heath-care contexts: embedding cinicay ed change in routine practice. Heath Serv Manage Res 2012;25: Martin GP, Weaver S, Currie G, Finn R, McDonad R. The Medium-Term Sustainabiity of Organisationa Change in the Nationa Heath Service: A Comparative Case Study of Cinicay Led Organisationa Innovations. Fina report. Southampton: NIHR Service Deivery and Organisation programme; Yin RK. Case Study Research: Design and Methods. Thousand Oaks, CA: SAGE Pubications; Gagnon YC. The Case Study as Research Method: A Practica Handbook. Québec City, QC: Presses de Université du Québec; Stake RE. The Art of Case Study Research. Thousand Oaks, CA: Sage Pubications; Woodside AG, Wison EJ. Case study research methods for theory buiding. J Bus Ind Mark 2003;18: Schuz K, Atman D, Moher D. CONSORT 2010 Statement: updated guideines for reporting parae group randomised trias. Ann Intern Med 2010;152: von Em E, Atman D, Egger M, Pocock S, Gotzsche P, Vandenbroucke J. The Strengthening the Reporting of Observationa Studies in Epidemioogy (STROBE) statement: guideines for reporting observationa studies. Ann Intern Med 2007;147: Moher D, Liberati A, Tetzaff J, Atman D, The PRISMA Group. Reporting items for systematic reviews and meta-anayses: the PRISMA statement. PLOS Med 2009;6:e /journa.pmed Gagnier J, Kiene G, Atman D, Moher D, Sox H, Riey D, et a. The CARE Guideines: Consensus-based cinica case reporting guideine deveopment. BMJ Case Rep 2013;7: Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 39

66 REFERENCES 16. O Brien B, Harris I, Beckman T, Reed D, Cook D. Standards for reporting quaitative research: a synthesis of recommendations. Acad Med 2014;89: ACM Wong G, Greenhagh T, Buckingham J, Pawson R. RAMESES pubication standards: reaist synthesis. BMC Med 2013; Wong G, Greenhagh T, Westhorp G, Buckingham J, Pawson R. RAMESES pubication standards: meta-narrative reviews. BMC Med 2013; Bossuyt P, Reitsma J, Bruns D, Gatsonis C, Gasziou P, Irwig L, et a. Towards compete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for Reporting of Diagnostic Accuracy. BMJ 2003;326: Davidoff F, Bataden P, Stevens D, Ogrinc G, Mooney S. Pubication guideines for quaity improvement in heath care: evoution of the SQUIRE project. Qua Saf Heath Care 2008;17:i Husereau D, Drummond M, Petrou S, Carswe C, Moher D, Greenberg D, et a. Consoidated Heath Economic Evauation Reporting Standards (CHEERS) statement. Eur J Heath Econ 2013;14: Byrne D. Compex reaist and configurationa approaches to cases: a radica synthesis. In Byrne D, Ragin C, editors. The SAGE Handbook of Case-Based Methods. London: SAGE Pubications; 2009 pp Byrne D. Introduction: case-based methods: why we need them; what they are; how to do them. In Byrne D, Ragin C, editors. The SAGE Handbook of Case-Based Methods. London: SAGE Pubications; 2009 pp Byrne DS. Compexity, configuration and cases. Theor Cut Soc 2005;22: / Fyvbjerg B, Landman T, Schram S. Important next steps in phronetic socia science. In Fyvbjerg B, Landman T, Schram S, editors. Rea Socia Science: Appied Phronesis. Cambridge: Cambridge University Press; pp Fyvbjerg B, Landman T, Schram S. Rea Socia Science: Appied Phronesis. Cambridge: Cambridge University Press; Fyvbjerg B. Case study. In Denzin NK, Lincon YS, editors. The SAGE Handbook of Quaitative Research. 4th edn. Thousand Oaks, CA: SAGE Pubications; pp Gomm R. Key Concepts in Socia Research Methods. London: Pagrave Macmian; Gomm R, Davies C. Using Evidence in Heath and Socia Care. London: SAGE Pubications; Gomm R, Hammersey M, Foster P. Case Study Method Key Issues, Key Texts. London: SAGE Pubications; Ragin CC. Refections on casing and case-oriented research. In Byrne D, Ragin CC, editors. The SAGE Handbook of Case-based Methods. London: SAGE Pubications; pp Ragin CC, Schneider G. Case-oriented theory buiding and theory testing. In Wiiams M, Vogt P, editors. The SAGE Handbook of Methodoogica Innovations. London: SAGE Pubications; pp Ragin CC, Schneider G. Comparative poitica anaysis: six case-oriented strategies. In Amenta E, Nash K, Scott A, editors. The New Backwe Companion to Poitica Socioogy. Chichester: Backwe; pp NIHR Journas Library

67 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO Yin RK. Appications of Case Study Research. 3rd edn. London: SAGE Pubications; Yin RK. Case study methods. In Cooper H, editor. APA Handbook of Research Methods in Psychoogy, Vo 2: Research Designs: Quantitative, Quaitative, Neuropsychoogica, and Bioogica. Washington, DC: American Psychoogica Association; pp Stake RE. Quaitative Case Studies. In Denzin NK, Lincon YS, editors. The SAGE Handbook of Quaitative Research. 3rd edn. Thousand Oaks, CA: SAGE Pubications; pp Custer R, Scarcea J, Stewart R. The modified Dephi technique a rotationa modification. J Voc Tech Educ 1999; Crowe S, Cresswe K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodo 2011;11: Darke P, Shanks G, Broadbent M. Successfuy competing case study research: combining rigour, reevance and pragmatism. Inf Syst J 1998;8: j x 40. Fitzgerad L, Dopson S. Comparative case study designs: their utiity and deveopment in organizationa research. In Buchanan DA, Brynam A, editors. The SAGE Handbook of Organizationa Research Methods. London: SAGE Pubications; pp Gibbert M, Ruigrok W. The what and how of case study rigor: three strategies based on pubished work. Organ Res Methods 2010;13: Gibbert M, Ruigrok W, Wicki B. What passes as a rigorous case study? Strategic Manage J 2008;29: Gigun JF. A case for case-studies in socia-work research. Soc Work 1994;39: Giham B. Case Study Research Method (Continuum Research Methods). London: Boomsbury Academic; Greene D, David JL. A research design for generaizing from mutipe case studies. Eva Program Pann 1984;7: Hays PA. Case study research. In demarrais K, Lapan SD, editors. Foundations for Research: Methods of Inquiry in Education and the Socia Sciences. Mahwah, NJ: Lawrence Erbaum Associates Pubishers; pp Hutchinson SA. The case study approach. In Moody LE, editor. Advancing Nursing Science Through Research (Vo 2). Newbury Park, CA: SAGE Pubications; pp Huws U, Dahmann S. Quaity Standards for Case Studies in the European Foundation. Dubin: European Foundation for the Improvement of Living and Working Conditions; Kaarbo J, Beasey RK. A practica guide to the comparative case study method in poitica psychoogy. Poit Psycho 1999;20: Meyer CB. A case in case study methodoogy. Fied Methods 2001;13: / X Moore TS, Lapan SD, Quartaroi MT. Case study research. In Lapan SD, Quartaroi MT, Riemer FJ, editors. Quaitative Research: An Introduction to Methods and Designs. San Francisco, CA: Jossey-Bass; pp Stake RE. Case studies. In Denzin NK, Lincon YS, editors. Handbook of Quaitative Research. Thousand Oaks, CA: SAGE Pubications; pp Thomas G. How to do your Case Study: A Guide for Students and Researchers. London: SAGE Pubications; Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 41

68 REFERENCES 54. Yin RK. The abridged version of case study research. Design and method. In Bickman L, Rog DJ, editors. Handbook of Appied Socia Research Methods. Thousand Oaks, CA: SAGE Pubications; pp Yin RK. Enhancing the quaity of case studies in heath services research. Heath Serv Res 1999;34: Yin RK. Case study methods. In Green GL, Camii G, Emore PB, editors. Handbook of Compementary Methods in Education Research. Mahwah, NJ: Lawrence Erbaum Associates; pp Attree M, Cooke H, Wakefied A. Patient safety in an Engish pre-registration nursing curricuum. Nurse Educ Pract 2008;8: Cooke H. The surveiance of nursing standards: an organisationa case study. Int J Nurs Stud 2006;43: Fied D, Reid D, Payne S, Ref M. Adut bereavement support in five hospices in Engand. Sheffied: University of Sheffied; Hawker S, Kerr C, Payne S, Seamark D, Davis C, Roberts H, et a. End-of-ife care in community hospitas: the perceptions of bereaved famiy members. Paiat Med 2006;20: Hutchinson S, Purce J. Managing ward managers for roes in HRM in the NHS: overworked and under-resourced. Hum Resour Manage J 2010;20: j x 62. Kyratsis Y, Ahmad R, Homes A. Understanding the Process of Innovation Adoption in 12 NHS Trusts Technoogy Seection, Procurement and Impementation to Hep Reduce HCAIs. London: Centre for Infection Prevention and Management, Imperia Coege London; Kyratsis Y, Ahmad R, Homes A. Technoogy adoption and impementation in organisations: comparative case studies of 12 Engish NHS Trusts. BMJ Open 2012;2:e /bmjopen Nationa Nursing Research Unit. The Productive Ward: Reeasing Time to Care. Learning and Impact Review. Coventry: NHS Institute for Innovation and Improvement; Payne S, Fied D, Ros L, Hawker S, Kerr C. Case study research methods in end-of-ife care: refections on three studies. J Adv Nurs 2007;58: j x 66. Payne S, Hawker S, Kerr C, Seamark D, Roberts H, Jarrett N, et a. Experiences of end-of-ife care in community hospitas. Heath Soc Care Community 2007;15: j x 67. Payne S, Kerr C, Hawker S, Seamark D, Davis C, Roberts H, et a. The Provision of Paiative Care for Edery Peope in Community Hospitas: Best Hope or Last Hope? Sheffied: Paiative and End-of-Life Care Research Group, University of Sheffied; Reid D, Fied D, Payne S, Ref M. Adut bereavement in five Engish hospices: types of support. Int J Paiat Nurs 2006;12: Reid D, Fied D, Payne S, Ref M. Adut bereavement in five Engish hospices: participants, organisations and pre-bereavement support. Int J Paiat Nurs 2006;12: /ijpn Ricketts T, Brooker C, Dent-Brown K. Menta heath in-reach teams in Engish prisons: aims, processes and impacts. Int J Prison Heath 2007;3: NIHR Journas Library

69 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO Robert G, Morrow E, Maben J, Griffiths P, Caard L. The adoption, oca impementation and assimiation into routine nursing practice of a nationa quaity improvement programme: the Productive Ward in Engand. J Cin Nurs 2011;20: j x 72. Ros L, Payne S. Chidhood bereavement services: issues in UK service provision. Mortaity 2004;9: The Offender Heath Research Network. A Nationa Evauation of Prison Menta Heath In-Reach Services. Manchester: The Offender Heath Research Network; Drennan VM, Hater M, Brearey S, Carneiro W, Gabe J, Gage H, et a. Investigating the contribution of physician assistants to primary care in Engand: a mixed-methods study. Heath Serv Deiv Res 2014;2(16). 75. McCourt C, Rayment J, Rance S, Sanda J. An ethnographic organisationa study of aongside midwifery units: a foow-on study from the Birthpace in Engand programme. Heath Serv Deiv Res 2014;2(7). 76. McDonad R. Evauation of the Advancing Quaity Pay for Performance Programme in the NHS North West. Southampton: NIHR Service Deivery and Organisation programme; Raine R, Waace I, Nic a Bháird C, Xanthopouou P, Lanceey A, Carke A, et a. Improving the effectiveness of mutidiscipinary team meetings for patients with chronic diseases: a prospective observationa study. Heath Serv Deiv Res 2014;2(37). 78. Raine R, Xanthopouou P, Waace I, Nic A Bháird C, Lanceey A, Carke A, et a. Determinants of treatment pan impementation in mutidiscipinary team meetings for patients with chronic diseases: a mixed-methods study. BMJ Qua Saf 2014;23: bmjqs Rande R, Greenhagh J, Hindmarsh J, Dowding D, Jayne D, Pearman A, et a. Integration of robotic surgery into routine practice and impacts on communication, coaboration, and decision making: a reaist process evauation protoco. Impement Sci 2014;9: / Rycroft-Maone J, Anderson R, Crane RS, Gibson A, Gradinger F, Griffiths HO, et a. Accessibiity and impementation in UK services of an effective depression reapse prevention programme mindfuness-based cognitive therapy (MBCT): ASPIRE study protoco. Impement Sci 2014;9: Waring J, Marsha F, Bishop S. Understanding the occupationa and organizationa boundaries to safe hospita discharge. J Heath Serv Res Poicy 2015;20(Supp. 1): / Waring J, Marsha F, Bishop S, Sahota O, Waker M, Currie G, et a. An ethnographic study of knowedge sharing across the boundaries between care processes, services and organisations: the contributions to safe hospita discharge. Heath Serv Deiv Res 2014;2(29). Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 43

70

71 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Appendix 1 Search strategies Library cataogue searches Heath Services Management Centre ONLINE (University of Birmingham) URL: Searched on: 14 Juy Records retrieved: 15. Search strategy case AND (method OR methods or methodoogy) Heath Management Onine (NHS Scotand) URL: Searched on: 14 Juy Records retrieved: 47. Search strategy Words= case AND W-subjects= method or methods or methodoogy and W-type= BK not JA The King s Fund ibrary database URL: Searched on: 14 Juy Records retrieved: 17. Advanced search Subject: case studies AND subject: research methods 10 records. Subject: case studies AND subject: methodoogy three records. Subject: case studies AND subject: methods 16 records. A 3 search strings were imited to books. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 45

72 APPENDIX 1 Key author searches Web searches via Googe were undertaken on 15 Juy 2014 to find ists of pubications for the five key authors identified. Pubications ists were found either by searching each author s institutiona website or, where that was not possibe, by using Googe Books and Googe Schoar. A pubications were scanned to identify those reating to case study methods. David Byrne five pubications. Bent Fyvbjerg 23 pubications. Roger Gomm 11 pubications. Chares Ragin 18 pubications. Robert K Yin 18 pubications. Database searches MEDLINE In-Process & Other Non-Indexed Citations and MEDLINE via OvidSP URL: Dates database searched: 1946 to 19 Juy Searched on: 22 Juy Records retrieved: 581. Search strategy 1. *Organizationa Case Studies/ (191) 2. Organizationa Case Studies/mt, st [Methods, Standards] (29) 3. (organi?ation$ adj5 case adj (study or studies)).ti,ab. (217) 4. 1 or 2 or 3 (415) 5. *Research Design/ (23,525) 6. *Methods/ (972) 7. 5 or 6 (24,381) 8. (case adj (study or studies)).ti. (24,261) 9. case study research.ti,ab. (149) 10. case-oriented research.ti,ab. (1) or 9 or 10 (24,358) and 11 (161) 13. case study method$.ti. (37) 14. case-based method$.ti,ab. (19) or 14 (56) or 12 or 15 (618) 17. exp animas/ not humans/ (3,968,668) not 17 (607) 19. imit 18 to engish anguage (581) 46 NIHR Journas Library

73 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Key / = indexing term (MeSH heading) exp = expoded MeSH heading * = focussed MeSH heading /mt, st [Methods, Standards] = MeSH heading restricted to those with Methods or Standards subheading appied $ = truncation? = widcard.ti,ab. = terms in either tite or abstract fieds adj5 = terms within five words of each other (any order) adj = terms next to each other (order specified) Appied Socia Sciences Index and Abstracts (ASSIA) via ProQuest URL: Search date: 22 Juy Records retrieved: 627. Search strategy (TI,AB(organi?ation* NEAR/5 ( case study OR case studies )) AND a.exact( Engish )) OR (((SU.EXACT ( Research methods ) AND a.exact( Engish )) OR (SU.EXACT( Methodoogy ) AND a.exact( Engish )) OR (SU.EXACT( Research design ) AND a.exact( Engish ))) AND ((SU.EXACT.EXPLODE( Case studies OR Singe case studies ) AND a.exact( Engish )) OR (TI,AB( case study OR case studies ) AND a.exact ( Engish )) OR (TI,AB( case study research ) AND a.exact( Engish )) OR (TI,AB( case-oriented research ) AND a.exact( Engish )))) OR (TI,AB( case study method* ) AND a.exact( Engish )) OR (TI,AB ( case-based method* ) AND a.exact( Engish )) Key SU.EXACT = subject heading TI,AB = terms in the tite or abstract fieds NEAR/5 = terms within five words of each other (any order) * = truncation = phrase search a.exact = anguage imit Heath Management Information Consortium via OvidSP URL: Dates database searched: 1979 to May Searched on: 22 Juy Records retrieved: 244. Search strategy 1. (organi?ation$ adj5 case adj (study or studies)).ti,ab. (78) 2. research strategies/ (33) 3. research design/ (198) 4. research methodoogy/ (287) Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 47

74 APPENDIX 1 5. research methods/ (1251) 6. methods/ (310) 7. method study/ (32) 8. evauation methods/ (120) 9. 2 or 3 or 4 or 5 or 6 or 7 or 8 (2114) 10. case studies/ (2901) 11. (case adj (study or studies)).ti,ab. (5541) 12. case study research.ti,ab. (48) 13. case-oriented research.ti,ab. (0) or 11 or 12 or 13 (7490) and 14 (120) 16. case study method$.ti,ab. (70) 17. case-based method$.ti,ab. (2) or 15 or 16 or 17 (262) 19. imit 18 to engish (244) Key / = indexing term $ = truncation? = widcard.ti,ab. = terms in either tite or abstract fieds adj5 = terms within five words of each other (any order) adj = terms next to each other (order specified) PsycINFO via OvidSP URL: Dates database searched: 1806 to Juy week 3, Searched on: 22 Juy Records retrieved: 856. Search strategy 1. (organi?ation$ adj3 case adj (study or studies)).ti,ab. (426) 2. *Methodoogy/ (21,666) 3. case study.md. (0) cc. (28,838) 5. 2 or 3 or 4 (43,481) 6. (case adj (study or studies)).ti. (19,192) 7. case study research.ti,ab. (768) 8. case-oriented research.ti,ab. (5) 9. 6 or 7 or 8 (19,729) and 9 (381) 11. case study method$.ti. (79) 12. case-based method$.ti,ab. (37) or 10 or 11 or 12 (887) 14. (rat or rats or mouse or mice or hamster or hamsters or anima or animas or dog or dogs or cat or cats or bovine or sheep).ti,ab,sh. (257,776) not 14 (883) 16. imit 15 to engish anguage (856) 48 NIHR Journas Library

75 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Key / = indexing term * = focussed subject heading.md. = terms in the methodoogy fied.cc. = cassification code (2260 is code for Research Methods and Experimenta Design).sh. = terms in subject heading fied $ = truncation? = widcard.ti,ab. = terms in either tite or abstract fieds adj3 = terms within three words of each other (any order) adj = terms next to each other (order specified) Socia Science Citation Index via Web of Science, Thomson Reuters URL: Dates database searched: 1956 to 18 Juy Search date: 22 Juy Records retrieved: Search strategy Indexes=SSCI Timespan=A years 1. TS=(organi?ationa NEAR/3 ( case study or case studies )) (454) 2. TS= case study research (895) 3. TS= case-oriented research (7) 4. TS= case study method* (969) 5. TS= case-based method* (40) 6. (#5 OR #3 OR #2 OR #1) AND LANGUAGE: (Engish) (1351) Key TS= topic tag; searches terms in tite, abstract, author keywords and keywords pus fieds * = truncation = phrase search NEAR/3 = terms within 3 words of each other (any order) Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 49

76

77 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Appendix 2 Synthesised data extraction of methodoogica texts Section Themes that cross mutipe sections Items Repication of case studies 8 Determinants of case study quaity 53 How we has the case been chosen? (Approach and processes to be adopted in data coection and anaysis) 53 Expanation and justification of the context for the study 53 Quaity of arguments being made and exporation of riva expanations 53 Pan Determine whether or not the case study is the appropriate method 48 Topic is new (i.e. itte quaitative or quantitative evidence) consider exporatory case studies 48 There is some quantitative evidence but itte is known about how or why consider expanatory case studies 48 Initia steps are: 44 Read reevant iterature 44 Get to know case/s in their setting 44 Decide what broad aims are 44 Defining/framing the research question(s) 8 Research question best for how and why questions 49 Both quantitative and quaitative evidence exists but there is a need by poicy stakehoders for information about current or best practice in specific contexts consider exampe case studies 48 Begin with a broad prima facie question, then refine using 53 Literature review 53 Storyboards/brainstorming/mind maps 53 Research questions 10 Issue questions or issue statements can be used to organise a case study. Issues identify one or more aspects of the situation or circumstance surrounding the case, in order to frame the inquiry 10 A number of research questions may be proposed at the beginning and refined with greater understanding of the case. Etic issues are brought in from the researcher from outside; emic issues emerge from inside the case. As the researcher begins to integrate etic and emic, the research question(s) evoves 10 One way to note the evoution of research question(s) is to retite the inquiry on a reguar (e.g. monthy) basis 10 Ensure that the focus has not shifted from the case to the issues 10 Progressive focusing : if eary research questions are not heping to thoroughy understand the case, or if new issues become apparent, the research questions can be changed 10 Thorough iterature review 8 Define the research as a case study 8 Identify the research question(s) 8 Carefuy formuated research question(s), informed by the existing iterature and a prior appreciation of the theoretica issues and setting(s) (aso references the work of Stake) 38 Make start on getting research questions into shape 44 Identify the origin of your seected case (subject): 53 Key case (good exampe; cassic or exempary case) 53 Outier case (showing something interesting because it is different from the norm) 53 Loca knowedge case (exampe chosen on the basis of persona experience) 53 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 51

78 APPENDIX 2 Section Items Identify the purpose(s) 53 Intrinsic 53 Instrumenta 53 Evauative 53 Expanatory 53 Exporatory 53 Identify the approach(s) 53 Testing a theory 53 Buiding a theory 53 Drawing a picture, iustrative 53 Descriptive 53 Interpretive 53 Experimenta 53 Identify the process(s) 53 Singe or mutipe 53 Nested 53 Parae 53 Sequentia 53 Retrospective 53 Snapshot 53 Diachronic 53 Organisations need to: 39 Be very cear about the research outcomes and how their organisation wi benefit from invovement 39 Ensure the researcher works with the organisation to identify what s in it for them 39 Participant organisations and participants need to know that adequate preparation for the study at that site has been carried out. 39 Design Define the starting point of the research to be done 48 Exporatory case studies 48 Define poicy reevance 48 Identify reevant stakehoders 48 If necessary narrow definition to ensure focus on poicy reevant aspects 48 Define other research components (e.g. expert interviews, iterature review, expert workshop) 48 Deveop draft form of words to describe need for indicators 48 Define required inputs for future events (e.g. conference, seminar, workshop) 48 Specify the need for recommendations 48 Define suitabe ength and pubication medium for reporting 48 Expanatory case studies Formuate cear research questions and/or hypotheses to be tested in the research 48 Referring to/anayse reevant quantitative data 48 Define how data wi be accessed and used 48 Describe data protection measures 48 Define required inputs for future events (e.g. conference, seminar, workshop) 48 Specify the need for recommendations 48 Define suitabe ength and pubication medium for reporting 52 NIHR Journas Library

79 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Section Items Exampe case studies 48 Deveop a ist of quaity criteria if concerned with good practice 48 Deveop a ist of seection criteria if concerned with iustrating the variety of practice 48 Define required inputs for future events (e.g. conference, seminar, workshop) 48 Decide whether a consoidated/synthesis report is required (i.e. whether to aow cases to speak for themseves or draw expicit essons) 48 Define the unit of anaysis and the ikey case(s) to be studied 8 Define the case, for exampe a group of neighbours vs. geographica neighbourhood 8 Bound the case, that is distinguish the subject of the case study (the phenomenon ) from externa data to the case (the context ). Spatia, tempora and other concrete boundaries shoud be considered. Abstractions (e.g. the concept of neighbouring ) cannot be considered a case 8 Unit of anaysis can be incident, event or event sequences 40 Unit of anaysis: another way to respond to researchers and respondents biases is to have more than one unit of anaysis in each case. This impies that, in addition to deveoping contrasts between the cases, researchers can focus on contrasts within the cases. In case studies, there is a choice of a hoistic or embedded design. A hoistic design examines the goba nature of the phenomenon, whereas an embedded design aso pays attention to subunit(s) 50 Use term conceptua framework but, simiar to Yin, state purpose of study, hypotheses or research questions and reasoning that ed to these. Define concepts. Describe construction of framework, for exampe iterature review and researcher experience 43 Seecting cases: in an intrinsic case study, the case is seected on its own merits. The case is seected not because it is representative of other cases, but because of its uniqueness. For an instrumenta case study, seecting a typica case can work we and aows investigation of an issue or phenomenon. In coective or mutipe case studies, a number of cases are carefuy seected. It is aso important to consider in advance the ikey burden and risks associated with participation for those who [or the site(s) which] comprise the case study 38 Seecting a case 10 For an instrumenta or coective case study, it is more important to seect a case that is informative for the study rather than necessariy representative of other cases. An informative case coud be typica or nove 10 Make some eary assessments of progress to see if the case shoud be dropped and another seected 10 Samping: the ogic in case studies invoves theoretica samping, in which the goa is to choose cases that are ikey to repicate or extend the emergent theory or to fi theoretica categories and provide exampes for poar types 50 When conducting a case study, there are severa important issues to decide when samping time: 50 how many times data shoud be coected? 50 when to enter the organisations 50 need to decide whether to coect data on a continuous basis or in distinct periods 50 Researcher shoud make expicit which of the variabes to be investigated are hypothesised to be most important for expaining the phenomenon 49 Important criteria for samping factors are that 45 they shoud be ascertainabe in advance (which usuay means from a distance as we) 45 their range of variation in the popuation of interest shoud be known 45 Seecting cases: choose comparabe cases (e.g. cuture, time period, etc.) 49 Need to choose cases carefuy to eiminate bias. Drawbacks with each strategy 49 Choose cases across popuation subgroups 49 Eisenhardt suggests that between 4 and 10 cases are desirabe for theory-buiding using case study research 39 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 53

80 APPENDIX 2 Section Items Practica issues that impact upon the design and scope of a case study research project, incuding: 39 the purpose for which the research is undertaken 39 the resources avaiabe to the researcher 39 the deiverabes required 39 potentia conficts between the needs and interests of sponsoring organisations and the requirements of the research objectives. Researchers must exercise judgement to ensure that an appropriate baance between these is maintained 39 Aso reduce bias by using mutipe sources of evidence 39 Deveop theory, propositions and reated issues to guide the anticipated case study and to generaise its findings. 8 Define the ogic inking the data to the propositions (i.e. anticipate what kind of anaytic techniques wi be used) 8 Define the criteria for interpreting the findings [i.e. expicity consider riva expanations (theories) at the outset, to guide decisions about which data shoud be coected this approach differs from methods such as ethnography and grounded theory] 8 A purey exporatory study without any initia propositions shoud state a purpose and the criteria by which the exporation wi be judged successfu or not 8 Theory-driven approach to defining case may hep generate knowedge that s transferabe to a range of contexts and behaviours and a more informed appreciation of how and why interventions worked or not 38 The conceptua framework shoud identify the main facts and events of interest in the subject of study and the main features of the context in which these facts and events are occurring 45 Identify the case study design (singe or mutipe, hoistic or embedded) 8 Four forms of mutipe case-study design based on different design ogic: 40 Matching or repication to expore or verify ideas 40 Comparison of difference to aid anaysis of reationships 40 Outiers; comparison of extremes to deineate key factors and shape of a fied 40 Embedded: to identify simiarities/differences within contexts 40 Decide on ongitudina or cross-sectiona approach 51 Choose to use singe or mutipe cases 50 The four main features of a mutipe case study design are: 45 a conceptua framework that provides the superordinate structure 45 a samping pan that ensures representativeness of the target popuation in the sampe of cases 45 procedures for the conduct of individua case studies that ensure sufficient comparabiity across cases 45 a cross-site anaysis strategy that tests the imiting conditions of the findings 45 Programme case study designs: identifying the specific programme to be investigated foowed by the seection of specific aspects that wi be thoroughy studied. Uness very sma and uncompicated, most programmes cannot be studied in their entirety 51 Test the design against four criteria for maintaining the quaity of a case study 8 Construct vaidity (identifying correct operationa measured for the concepts being studied) 8 Interna vaidity (seeking to estabish a causa reationship, whereby certain conditions are beieved to ead to other conditions, as distinguished from spurious reationships). For expanatory or causa studies ony, not appicabe to exporatory or descriptive studies 8 Externa vaidity (defining the domain to which a study s findings can be generaised). Anaytic generaisation using theory is most appicabe to singe case studies 8 Reiabiity (demonstrating that the operations of a study can be repeated with the same resuts). Case study protoco and deveopment of case study database 8 54 NIHR Journas Library

81 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Section Items Sees probems with Yin quaity criteria of construct, interna, externa vaidity and reiabiity 40 Consideration of the appropriateness of methods used for the question and subject matter and why it was that quaitative methods were appropriate 40 Adequacy of samping and expanation of samping strategy 40 Rigour of data anaysis (was it conducted in a systematic way and was it successfu in incorporating a observations and deaing with variation) 40 Refexivity of account Sensitivity to the ways in which the researcher and research process have shaped the data coection and provision of sufficient information of research process for readers to judge 40 Adequacy of presentation of findings is it cear how anaysis fows from data and are sufficient data presented to justify concusion 40 Worth and reevance of that research 40 Statistica concusion vaidity concerned with whether context intervention takes pace, is understood and is described, with use of reiabe and vaid instruments and with appropriate statistics 43 Construct vaidity, externa vaidity and interna vaidity 43 Need a diaogue among investigators to construct a chain of evidence. Greatest threat to vaidity arises from a faiure to consider aternatives exhaustivey or to incude a reevant variations in the sampe of cases. Need to consut with a range of experts with diverse points of view, both during the fina stages of deveoping the conceptua framework and after drafting concusions. These procedures need to be buit into overa method of approach to perhaps provide a reaistic means of increasing the vaidity 45 Prepare Hone skis as a case study researcher (ask good questions, be a good istener, stay adaptive, have a firm grasp on the issues being studied, avoid biases and conduct research ethicay) 8 Researcher shoud write down expectations and preferences to be abe to detect own bias 44 Researchers shoud prepare themseves with sufficient background information about a case study site prior to commencing data coection 39 Train for specific case study 8 Deveop case study protoco with four main sections: 8 Overview of the case study (objectives and auspices, case study issues and reevant topic readings) 8 Data coection procedures (ethica consideration, identify ikey sources of data, presentation of credentias to fied contacts and other ogistica reminders) 8 Data coection questions (specific research questions and the potentia sources of data for each question) 8 A guide for the case study report (outine, format for the data, use and presentation of other documentation, bibiographica information) 8 Deveop the research specification 48 Describe the unit of anaysis 48 Propose the number and distribution of cases (if more than one) 48 Deveop a reaistic timetabe 48 Finaise proposa specification 48 Deveop the proposa Briefy summarise existing knowedge 48 Provide a rationae for the seection of case(s) 48 Summarise methodoogy (research instruments, access, obtaining informed consent, interview methods, record keeping, anaysis and interpretation) 48 Propose a management pan (identify key performance indicators and miestones) 48 Summarise risk anaysis (e.g. identify troube-shooting mechanisms) 48 Summarise timetabe 48 Summarise costs 48 Describe deiverabes 48 Prepare CVs 48 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 55

82 APPENDIX 2 Section Items Summarise key research questions and/or hypotheses 48 Screen candidates and seect fina cases (where there is a number of potentiay eigibe cases to study) 8 Conduct piot case study 8 Gain approva for human subjects protection 8 Essentia parts of a data-gathering pan: 10 Definition of case 10 List of research questions 10 Identification of hepers 10 Data sources 10 Aocation of time and expenses 10 Intended reporting 10 Choose vaid data coection too (e.g. questionnaire or observation protoco) 51 Piot test too used for data coection 51 Fied test too on participants simiar to the actua participants who wi be studied 51 Deveop observation pan 51 Need to consider the interpreters of the data researchers wi bring their own perspectives and biases 43 To impement a mutipe case study design propery, it is typicay necessary to train a group of data coectors to think and act more or ess aike 45 Coect Consider six sources of evidence: 8 Documentation 8 Archiva records 8 Interviews be aware of bias, reca, and inaccurate articuation (corroborate with other sources) 8 Direct observations 8 Participant-observation 8 Physica artefacts 8 Document review seection guided by the research question(s) 10 Observation 10 Keep a good record of events during observation to provide a reativey incontestabe description for further anaysis and reporting 10 Interview requires a strong advance pan and pioting 10 Prior to the study, the case study researcher aso chooses a method for recording information from interviews and observations, for exampe videotape, audiotape or note taking 51 Four principes of data coection (hep estabish construct vaidity and reiabiity among the sources of data) 8 Trianguate evidence from different sources 8 data trianguation 8 investigator trianguation 8 theory trianguation 8 methodoogica trianguation 8 Assembe data into a comprehensive case study database 8 The data or evidentiary database, in which the raw data can be inspected (incuding notes, documents, tabes and narratives) 8 The researcher s report (in artice, book or ora form) 8 Maintain chain of evidence 8 Increase reiabiity by aowing an externa observer to foow the derivation of any evidence from initia research questions to utimate case study concusions 8 Case study report case study database citations to specific evidentiary sources within the database case study protoco case study questions 8 Exercise care in using data from eectronic sources, for exampe cross-check the accuracy of onine sources, especiay information from socia media sites 8 56 NIHR Journas Library

83 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Section Items Obtain access and permissions 10 Description of contexts (physica, economic, historica, cutura, aesthetic) 10 May continue to search for data unti saturation is reached; that is, the evidence becomes redundant, with no new information 51 Uses Stake s checkist to assess quaity of a case study report 38 It is important that data sources from different cases are, where possibe, broady comparabe even though they may vary in nature and depth 38 Researchers shoud maintain a og of evidence and persona notes whie conducting the research and coecting data to be used as part of the overa database 44 The roe of history in understanding current strategy, choices and eves of institutionaisation are often overooked and shoud be considered during data coection 40 Need to deveop empathy between researcher and subjects and understand power dynamics within setting 46 Construct a case codebook to guide coection of evidence for the variabes in the study 49 Record and report the way data are coected 49 Contextua detai unit of anaysis rarey isoated from and unaffected by environmenta factors; need to describe context in detai to understand and interpret 43 Refers to ecosystem framework with notions of mutipe, interacting contextuaised systems 43 A the above (Data coection) shoud be reported as part of methods section 43 Anayse Before anaysis researchers need to famiiarise themseves with the data 40 Revea researcher position. If the researcher has a cose reationship or a past history with the case being studied, this information shoud be made transparent 51 Researcher biases or predispositions can be made expicit in a bracketed interview prior to the study. The researcher and case study audiences must examine more carefuy any resuts that match the researcher s preconceived expectations. 51 Two types of researcher bias may be recognised: the effects of the researcher on events and the behaviour of participants at the case study site, and the researcher s own beiefs, vaues and prior assumptions 39 Array and dispay data in different ways 8 Watch for promising patterns, insights and concepts 8 Code data: When the researcher sees simiarities between various components, these components wi be assigned the same category or code 51 Assign conceptua categories to words (or signs), which represent aspects of the particuar theory being investigated. The importance of a concept is reated to the frequency with which it occurs 39 Deveop a genera anaytic strategy (or mutipe strategies): 8 Reying on theoretica propositions (i.e. the propositions on which the origina objectives and design of the case study were based) 8 Working your data from the ground up. Contrasts directy with the preceding approach. Use an inductive strategy, poring through your data, deveoping concepts and reationships between concepts as you do so. Basis of the grounded theory approach 8 Deveoping a case description; i.e. organise the case study according to some descriptive framework (as opposed to an expanatory theory). The description may ater hep to identify the appropriate expanation to be anaysed 8 Examining pausibe riva expanations. May work in combination with the above three strategies. Distinguishes between craft rivas and rea-word rivas. Riva expanations shoud be anticipated before even coecting data 8 Craft rivas Nu hypothesis (observation soey due to chance) 8 Threats to vaidity (e.g. instrumentation, regression seection) 8 Investigator bias (e.g. experimenter effect, reactivity in fied research) 8 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 57

84 APPENDIX 2 Section Items Rea-word rivas Direct riva (e.g. resuts due to intervention 2, not intervention 1) 8 Co-minged riva (e.g. intervention 1 pus one or more other interventions contributed to the resuts) 8 Impementation riva (resuts due to the impementation process, rather than the substantive intervention) 8 Riva theory (a theory different from the origina theory expains the resuts better) Super riva (a force arger than but incuding the intervention accounts for the resuts) 8 Societa riva (socia trends, not any particuar force or intervention account for the resuts) 8 Aong with the genera strategy, consider five anaytic techniques: 8 Pattern matching. If empiricay based patterns appear simiar to predicted patterns, the resuts can strengthen interna vaidity. Especiay true if a pattern of resuts for a number of different outcomes has been predicted correcty ( non-equivaent, dependent variabes design ). May further strengthen this through theoretica repication or itera repication across studies. Need to acknowedge possibe threats to vaidity (e.g. confounding variabes) and show that these cannot account for the patterns observed 8 Expanation buiding, i.e. stipuating a presumed set of causa inks about a phenomenon or how or why something happened. Likey to be an iterative process, in which an initia expanatory proposition is compared against the findings of a case, revised if necessary, then compared against other detais of the case, and repeated as many times as needed. However, there is a risk of drifting from the origina research question or introducing bias; suggested safeguards are frequenty checking the origina purpose, empoying critica friends and examining aternative expanations 8 Time-series anaysis. Specificay ooking at empirica trend(s) over time for a dependent variabe and comparing this empirica trend with one or more theoretica predictions. Like pattern matching, but expicity invoving statistica techniques. Simpe time series might invove a inear trend for a singe dependent variabe; more compex series might invove non-inear trends and/or mutipe variabes. The researcher must identify the specific indicator(s) to be traced over time, the time intervas to be covered and the presumed reationships among events prior to coecting the actua data 8 Logic modes: describe a repeated cause-and-effect sequence of events inked together (i.e. intervention/phenomenon immediate outcome intermediate outcome utimate outcome). Provides an initia hypothesis about the case and then provides a framework for anaysing the data. Can use quantitative, quaitative or both kinds of data. The need to consider the infuence of rea-word and other contextua conditions wi vary between studies 8 Cross-case synthesis. Appies ony to mutipe cases. Synthesising two or more independent cases can be more robust than having just a singe case. Empirica data from mutipe cases coud be used to examine a theory or be combined statisticay for precision (i.e. meta-anaysis) 8 Stick to four principes of good socia science research: 8 Attend to a the evidence 8 Address a riva expanations and interpretations 8 Anaysis shoud address the most significant aspect of the case study (not digress too far into esser issues) 8 Use your own prior, expert knowedge 8 Generaisation of resuts 8 Identifying and considering riva expanations 8 Categorica aggregation vs. direct interpretation 10 The former appears to mean ooking for repeated observations before making an interpretation, the atter making an interpretation about a specific observation 10 Correspondence and patterns 10 Patterns may foow from research questions or emerge from the anaysis NIHR Journas Library

85 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Section Items Naturaistic generaisations aow the reader to make generaisations by providing them with the opportunity for vicarious experience aongside the researcher s own interpretation(s) 10 Incude accounts of matters that readers are aready famiiar with so they can gauge the accuracy, competeness of reports of other matters 10 Provide enough raw data prior to interpretation for readers to consider their own aternative interpretations 10 Describe in pain anguage how trianguation was carried out, especiay in confirming and disconfirming major assertions 10 Make data avaiabe on the researcher and other sources of input 10 Incude the reactions of data sources (and other prospective readers) to the accounts 10 Trianguation 10 Vaidation meaning may be ascribed to a particuar observation, but mutipe observations give us grounds for revising our interpretation 10 Targets for trianguation there wi be a greater need for trianguation in the case of more dubious or contested descriptions and for key interpretations 10 Trianguation protocos 10 Data source trianguation an effort to see if what we are observing and reporting carries the same meaning when found under different circumstances 10 Investigator trianguation have other researchers ook at the same scene or phenomenon 10 Theory trianguation may invove two investigators with different theoretica viewpoints 10 Methodoogica trianguation using different methodoogica approaches to examine the same phenomenon 10 Member checking ask actors to review the materia for accuracy and paatabiity 10 Conduct appropriate data anayses; examining researcher preparation and bias; member checking (reviewing draft findings by key informants to see if they affirm the vaidity of the report); undertaking an externa review and interpretation to improve the vaidity and trustworthiness of case study findings 51 In coective case studies, it is hepfu to anayse data reating to the individua component cases first, before making comparisons across cases 38 The Framework approach is a practica approach, consisting of five stages (famiiarisation; identifying a thematic framework; indexing; charting; and mapping and interpretation) to manage and anayse arge datasets, particuary if time is imited 38 Each month shoud do major review of progress and write progress report for researcher records and others on what achieved and how design and theory have deveoped 44 Look for discrepant data evidence that compicates emerging understanding 44 Check representativeness of data a shades of opinion 44 Check ideas and expanations with those in the cuture (e.g. organisation) 44 Need to buid theories and examine negative evidence 44 Need to be refective and have feedback workshops with onsite coaborators to road test eary formuations 40 Decide whether to adopt a framework for anaysis or adopt a grounded approach 40 Focus on research questions during anaysis. Remember aim is not to gain a compete picture of the site but to answer the research questions 46 One approach in examining vaidity and reiabiity is to appy the criteria used in quantitative research, e.g. objectivity/intersubjectivity, construct vaidity, interna vaidity, externa vaidity, and reiabiity 50 The basic issue of objectivity can be framed as one of reative neutraity and reasonabe freedom from unacknowedged research biases (Mies and Huberman, 1994). One way to guard against this bias is for the researcher to expicity recognise his or her presuppositions and to make a conscious effort to set these aside in the anaysis (Gummesson, 1988). Furthermore, riva concusions shoud be considered 50 Construct vaidity can be strengthened by appying a ongitudina muticase approach, trianguation and use of feedback oops. Gives opportunity to test sensitivity of construct measures to the passage of time 50 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 59

86 APPENDIX 2 Section Items Interna vaidity concerns the vaidity of the postuated reationships among the concepts needs to be open to scrutiny 50 Generaisabiity: The vaidity of the extrapoation depends not on the typicaity or representativeness of the case but on the cogency of the theoretica reasoning. One way to increase the generaisabiity is to appy a muticase approach (p. 347) 50 The interpretive researcher is presenting their interpretation of other peope s interpretations 39 Interpretation aims to make sense of the object of study by iterating between understanding of the object as a whoe and understanding of its parts 39 Generaisation demonstrated through showing the inkages between findings and previous knowedge. Use anaytic generaisation, not probabiistic type 43 Theory deveopment 43 Process-reevant case studies focus on what happened, on how intervention worked and what major actors in the impementation process did. Iuminate outcomes by showing the practica activities and steps eading to overa impact of intervention 43 Outcome-oriented case studies focus on whether change occurred and whether it is attributed to intervention 43 The first step of the cross-site anaysis is to generate a working set of propositions (findings from the individua cases restated so as to appy, in principe, to a the cases) 45 Transate various findings into statements that are subject to empirica confirmation or disconfirmation 45 Organise the propositions by topics and subtopics to the extent that the structure of the interview/debriefing guide corresponds we to the reaity encountered in the fied 45 Having generated and organised the propositions, need to test each one against each reevant case 45 Through case-by-case comparisons, the anayst fine-tunes, modifies and quaifies the propositions so that they express precisey the imiting conditions reveaed by the pattern of findings across a cases 45 If the amount of modification required to make a proposition hod in a instances is excessive amounting to a site-dependent phenomenon, it is dropped from the cross-site anaysis 45 After the modified propositions are organised into cumps directed to particuar research questions, the findings shoud be communicated ceary with carefuy chosen exampes. Need to differentiate ceary in the report what can and cannot be generaised 45 Threats to vaidity in case study research may be cassified into two broad types of potentia errors in inference: 45 those resuting from a faiure to check out aternative expanatory patterns exhaustivey 45 those resuting from a faiure to achieve representative heterogeneity of important expanatory factors among respondents interviewed 45 The evauator needs to offer some evidence to the audience that the heterogeneity of the sampe of cases is representative of the heterogeneity of the target popuation 45 Most of the expanatory patterns that constitute the resuts of a mutipe case study evauation can be restated in the form of empiricay testabe propositions 45 Share Define audience, whether for written or ora compositions 8 Consider the most appropriate overa reporting structure 8 Linear-anaytic 8 Comparative 8 Chronoogica 8 Theory-buiding 8 Suspense 8 Unsequenced 8 60 NIHR Journas Library

87 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Section Items The traditiona report of statement of probem, review of iterature, design, data gathering, anaysis and concusions is particuary i-fitting for a case study report. The case is not a probem or a hypothesis. A report is ikey to foow the sequence in Stake s exampe above, or foow one of these paths: 10 A chronoogica or biographica deveopment of the case 10 A researcher s view of coming to know the case 10 Description one by one of severa major components of the case 10 Write case study reports in an agreed format, incuding iustrative quotes 48 When reporting case studies, foow a recognised case study reporting structure that has been used in pubished case study research iterature within the fied 39 The case study report must be compete and must contain sufficient evidence to support the findings 39 Secondary data that is not essentia for understanding and evauating the case study anaysis and concusions shoud be omitted 39 Presentation of data in tabuar form is often a usefu means of summarising and compressing data 39 The overa goas in writing up case studies are to adopt a cear and ucid writing stye and to present the critica evidence judiciousy and effectivey 39 Starting eary, compose textua and visua materias 8 Organising the report eary set out broad sections or chapters to begin. Exampe given by Stake: 10 Entry vignette 10 Issue identification, purpose and method of study 10 Extensive narrative description to further define case and contexts 10 Deveopment of issues 10 Descriptive detai, documents, quotations, trianguating data 10 Assertions 10 Cosing vignette 10 Vignettes temptation to seect atypica, rare, and vivid moments mosty because they coincide with the researcher s prediections needs to be chaenged 10 The content of case study reports can vary 51 Descriptive: provides a detaied account of what is happening in a particuar programme 51 Interpretive: if the report adds expanation in addition to description; for exampe, expaining why the programme is impemented in a particuar way 51 The goa of a case study report is to use description to provide the reader with a vicarious experience, or a sense of being there in person, and to enabe understanding of the experience from the informants perspectives. 51 Think about narrative dramaturgicay, i.e. think in terms of actors, roes and stages 53 Methodoogy section shoud address: 8 Overa tone (thoughtfu, baanced and transparent) 8 Research questions (shoud be dominated by how and why questions) 8 Design 8 Definition of case(s) and how seected 8 The (ogica) connection between the research question(s) and the data to be coected Rivas that were considered 8 Overview of rest of methodoogy section (brief summary of data coection and anaysis methods, to aow the reader to skip the subsequent detais if they wish) 8 Data coected 8 Emphasis on how the data provided an up-cose and in-depth coverage of the case 8 Presentation of the case study protoco and how it was used 8 List of sources in order of importance; further detais about specific items within each source 8 How the data were verified (i.e. trianguation methods) 8 Unexpected difficuties and how they might have affected the data coection 8 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 61

88 APPENDIX 2 Section Items Anaysis methods 8 Description of the anaytic approach, for exampe pattern matching, expanation buiding, etc. 8 Identification of software and how used 8 Caveats about study 8 Inherent shortcomings in the design and anaysis and how these might have infuenced the findings 8 Dispay enough evidence for the reader to reach their own concusions 8 Case study investigators can greaty increase the face vaidity of their concusions by preserving a chain of evidence concerning the basis on which their decisions in the process of constructing expanatory patterns were made 45 Review and re-compose unti done we 8 Where possibe have informants/participants review the draft report 8 Readers reader reasoning shoud be assisted in the way the report is written, by maximising the reader encounter with the compexity of the case. Try to anticipate what vicarious experiences wi do for the reader, and organise the manuscript in a way that faciitates naturaistic generaisation 10 Critique checkist for a case study report: 10 Is this report easy to read? 10 Does it fit together, each sentence contributing to the whoe? 10 Does this report have a conceptua structure (i.e. themes or issues)? 10 Are its issues deveoped in a serious and schoary way? 10 Is the case adequatey defined? 10 Is there a sense of story to the presentation? 10 Is the reader provided with some vicarious experience? 10 Have quotations been used effectivey? 10 Are headings, figures, artefacts, appendices, indexes effectivey used? 10 Was it edited we, then again with a ast poish? 10 Has the author made sound assertions, neither over- nor underinterpreting? 10 Has adequate attention been paid to various contexts? 10 Were sufficient raw data presented? 10 Were data sources we chosen and in sufficient number? 10 Do observations and interpretations appear to have been trianguated? 10 Is the roe and point of view of the researcher nicey apparent? 10 Is empathy shown for a sides? 10 Are persona intentions examined? 10 Does it appear individuas were put at risk? 10 Drawing on case study reports and, where reevant, summary sheets, interpret case study resuts 48 Revise report, taking account of feedback from stakehoders 48 Prepare any other deiverabes 48 Pubish report 48 Disseminate to scientific (exporatory and expanatory case studies) and poicy audiences (exporatory and exampe case studies) 48 Need to provide researcher s perspective and reationship to the case(s). Audience needs to understand researcher s roe and perspective to accept findings NIHR Journas Library

89 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Appendix 3 Dephi consensus process round 1 questionnaire Dephi consensus process round 1 questionnaire Note: Direct output from Quatrics survey software Reporting standards for organisationa case studies: round one Thank you for taking part in this Dephi exercise which wi run over a period of 3 weeks and require you to compete two rounds of questions. This first questionnaire shoud take about 30 minutes to compete, and responses shoud be submitted by 5pm (UK time) on Monday 16th February. The aim of the exercise is to deveop a minimum set of standards to improve the quaity and consistency of reporting of organisationa case studies. For the purposes of this exercise, we have defined this as any case study focused on an organized body of peope with a particuar purpose, such as a business, government department, charity, etc (as opposed to a case study of individuas). The resuts wi be coated and circuated with the second round of the exercise about two weeks after cosure of the first round. The second round is ikey to require fewer responses and therefore take ess time to compete. Your continued participation woud be greaty appreciated in order to achieve as cear a consensus as possibe. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 63

90 APPENDIX 3 In this first round, we wi present you with a unique items identified from a review of the methodoogica iterature. Each item is foowed by one or more numbered references e.g. (1,3,7). These refer to the origina source of the item - usuay a methodoogica text. Source detais are provided at the end of the survey. We have made the assumption that some form of reporting standard is both possibe and desirabe, so emphasis has been paced on practica suggestions rather than more abstract or theoretica issues. Items have been de-dupicated and grouped under headings for ease of rating. We have tried to avoid making judgements about the vaue of individua items, since this is the objective of the Dephi consutation. You are asked to indicate your persona preferences for each item, by rating it as Essentia, Desirabe, or Not necessary. If you beieve an item is absoutey necessary when reporting an organisationa case study, pease rate it as "Essentia". Items that you consider usefu but not essentia shoud be marked as "Desirabe". If you consider an item to be unnecessary, uncear, redundant, or not particuary meaningfu, pease rate it as "Not necessary". After rating the existing items, you wi be given the opportunity to suggest any additiona essentia items, as we as comment on the structure and grouping of items presented here. Describing the design (Section 1 of 7) Pease rate how important it is to incude the foowing items when reporting the design of the organisationa case study Define the research as a case study(1) Essentia Desirabe Not necessary Describe why case study is the appropriate method(2) Essentia Desirabe Not necessary Define the poicy reevance(2) Essentia Desirabe Not necessary 64 NIHR Journas Library

91 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 State the broad aims of the study(7) Essentia Desirabe Not necessary Identify the purpose of the case study(1, 4) e.g. Exporatory: The topic is new (i.e. itte quaitative or quantitative evidence)(2, 6) Expanatory: There is some quantitative evidence but itte is known about how or why aspects(2, 8, 6) Intrinsic: The case is seected on its own merits. The case is seected not because it is representative of other cases, but because of its uniqueness(5, 6) Instrumenta / Exampe: Seecting a typica case that aows investigation of an issue or phenomenon(5, 6) Both quantitative and quaitative evidence exists but there is a need by poicy stakehoders for information about current or best practice in specific contexts (2) Evauative: Evauation of the impact of practice or intervention(6) Essentia Desirabe Not necessary Identify the broad approach(es) e.g. Testing a theory(6); Buiding a theory(6); Drawing a picture/iustrative(6); Descriptive(6); Interpretive(6); Experimenta(6) Essentia Desirabe Not necessary Identify the process(es) (6) State whether it is a singe or mutipe/coective case study(1, 6, 9) (5, 10), aong with any other design characteristics e.g. Embedded/Nested(1, 6); Parae(6); Sequentia(6); Retrospective(6); Cross-sectiona / Snapshot(3, 6); Longitudina / Diachronic(3, 6) Essentia Desirabe Not necessary Define the case broady e.g. in a case study of neighbouring the case might be defined as either a group of neighbours (peope) or as a geographica neighbourhood (pace)(1) Essentia Desirabe Not necessary Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 65

92 APPENDIX 3 Identify the specific case(s)(1, 5) and justify the seection(5, 6) e.g. Key case (good exampe; cassic or exempary case) (6); Outier case (showing something interesting because it is different from the norm) (6); Loca knowedge case (exampe chosen on the basis of persona experience) (6) Essentia Desirabe Not necessary Describe the boundaries of the case i.e. distinguish the subject of the case study (the phenomenon ) from externa data to the case (the context ). Spatia, tempora, and other concrete boundaries shoud be considered. Abstractions (e.g. the concept of neighbouring ) cannot be considered a case. (1) Essentia Desirabe Not necessary Describe setting/context (physica, economic, historica, cutura, aesthetic) surrounding the case(5, 7) Essentia Desirabe Not necessary Mention any riva cases that were considered(1) Essentia Desirabe Not necessary Describe the ikey burden and risks associated with participation for those who (or the site(s) which) comprise the case study(11) Essentia Desirabe Not necessary 66 NIHR Journas Library

93 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Offer some evidence to the audience that the heterogeneity of the sampe of cases is representative of the heterogeneity of the target popuation(9) Essentia Desirabe Not necessary Describe some eary assessments of progress to see if the case shoud be dropped and another seected(5) Essentia Desirabe Not necessary State the research question(s)/hypotheses(1) Essentia Desirabe Not necessary Describe how the fina research question(s) was deveoped and refined from the broad prima facie question(s)(2, 5, 6, 7) Essentia Desirabe Not necessary Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 67

94 APPENDIX 3 Rate the importance of the foowing toos and techniques for describing deveopment of the fina research question Literature review(6) Storyboards / brainstorming / mind maps(6) A prior appreciation of the theoretica issues and setting(s)(11) "Issue questions or issue statements. ( Issues identify one or more aspects of the situation or circumstance surrounding the case, in order to frame the inquiry)(5) Resoution of etic and emic issues. (Etic issues are brought in from the researcher from outside; emic issues emerge from inside the case. As the researcher begins to integrate etic and emic, the research question(s) evoves)(5) Retiting the inquiry on a reguar (e.g. monthy) basis in order to note the evoution of the research question(s)(5) Progressive focusing : if eary research questions are not heping to thoroughy understand the case, or if new issues become apparent, 68 NIHR Journas Library

95 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 change the research questions(5) State the deiverabes required(4) Essentia Desirabe Not necessary State the impications of the resources avaiabe to the researcher(4) Essentia Desirabe Not necessary Acknowedge the potentia conficts between the needs and interests of any sponsoring organizations and the requirements of the research objectives. Show judgment to ensure that an appropriate baance between these is maintained(4) Essentia Desirabe Not necessary Specify the need for recommendations(2) Essentia Desirabe Not necessary Present the case study protoco and describe how it was used(1) Essentia Desirabe Not necessary Do you have any other comments about the design section? (an opportunity to add more items wi be given ater in this survey) Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 69

96 APPENDIX 3 Background, context, and theory (Section 2 of 7) Pease rate how important it is to incude the foowing items when reporting the background, context and theory of an organisationa case study Report the findings of a thorough iterature review(1, 7) Essentia Desirabe Not necessary Describe any other preparatory research components (e.g. expert interviews, expert workshop)(2) Essentia Desirabe Not necessary Report whether a piot case study has been conducted(1) Essentia Desirabe Not necessary Describe the theory, propositions and reated issues deveoped to guide the case study and to generaise its findings(1) Essentia Desirabe Not necessary 70 NIHR Journas Library

97 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Rate the importance of the foowing techniques for describing the deveopment of theory, propositions and issues Outine the conceptua structure (i.e. themes or issues)(5) The conceptua framework shoud identify the main facts and events of interest in the subject of study and the main features of the context in which these facts and events are occurring(9) Outine the (ogica) connection between the research question(s) and the data coected(1) Define the ogic inking the data to the propositions (i.e. what kind of anaytic techniques were used)(1) Define the criteria for interpreting the findings (i.e. expicity consider riva expanations (theories) at the outset, to guide decisions about which data shoud be coected, uness using grounded theory)(1) For purey exporatory studies without any initia propositions, state a purpose and the criteria by which the exporation is judged Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 71

98 APPENDIX 3 successfu or not(1) State which of the variabes being investigated are hypothesized to be most important for expaining the phenomenon(8) Describe whether a range of experts were consuted during the fina stages of deveoping the conceptua framework and report the findings of this consutation(9) Do you have any other comments about the background, context and theory section? (an opportunity to add more items wi be given ater in this survey) Describing the data coection (Section 3 of 7) Pease rate how important it is to incude the foowing items when reporting the data coection Describe how data were coected(1) Essentia Desirabe Not necessary Describe whether the data provided an up cose and in-depth coverage of the case(1) Essentia Desirabe Not necessary 72 NIHR Journas Library

99 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Describe the sources of evidence used(1, 5) e.g. Documentation(1, 5); Archiva records(1); Interviews(1) (5); Direct observations(1, 5); Participant-observation(1); Physica artefacts(1) Essentia Desirabe Not necessary List evidence sources in order of importance; give further detais about specific items within each source(1, 4) Essentia Desirabe Not necessary State that a the evidence was examined(1) Essentia Desirabe Not necessary Describe the data coection too(s) (e.g. questionnaire or observation protoco), incuding a description of any pioting or fied testing of the too(3, 5) Essentia Desirabe Not necessary State whether a comprehensive case study database, in which the raw data can be inspected (incuding notes, documents, tabes and narratives) is avaiabe to readers(1) Essentia Desirabe Not necessary Describe data protection measures(2) Essentia Desirabe Not necessary Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 73

100 APPENDIX 3 Describe any ethica considerations and obtainment of reevant approvas, access and permissions(1) Essentia Desirabe Not necessary Describe the observation pan and how it was deveoped(3) Essentia Desirabe Not necessary Search for data unti saturation is reached, that is, the evidence becomes redundant, with no new information(3) Essentia Desirabe Not necessary Describe how the data were coded(3, 4) Essentia Desirabe Not necessary Describe the ikey impact of the researcher on events and the behaviour of participants at the case study site, and the researcher's own beiefs, vaues and prior assumptions(4, 12) Essentia Desirabe Not necessary Do you have any other comments about the data coection section? (an opportunity to add more items wi be given ater in this survey) Describing the data anaysis (Section 4 of 7)Pease rate how important it is to incude the foowing items when reporting the anaysis of an organisationa case study 74 NIHR Journas Library

101 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Describe the anaysis methods(1) Essentia Desirabe Not necessary Structure the reporting of the anaysis around the research questions(13) Essentia Desirabe Not necessary State whether an inductive (e.g. grounded) or deductive (e.g. hypothesis testing / theoretica framework) approach to the anaysis has been taken(1, 10, 14) Essentia Desirabe Not necessary In coective case studies, anayse data reating to the individua component cases first, before making comparisons across cases(11) Essentia Desirabe Not necessary Describe the anaytic approach in detai(1) e.g. Pattern matching. If empiricay based patterns appear simiar to predicted patterns, the resuts can strengthen interna vaidity. May further strengthen through theoretica repication or itera repication across studies. Need to acknowedge possibe threats to vaidity (e.g. confounding variabes) and show that these cannot account for the patterns observed.(1) Patterns may foow from research questions or emerge from the anaysis(5) Expanation buiding i.e. stipuating a presumed set of causa inks about a phenomenon or how or why something happened. Likey to be an iterative process, in which an initia expanatory proposition is compared against the findings of a case, revised if necessary, then compared against other detais of the case, and repeated as many times as needed. However, there is a risk of drifting from the origina research question or introducing bias; suggested safeguards are frequenty checking the origina purpose, empoying critica friends, and examining aternative expanations.(1) Categorica aggregation versus direct interpretation - the former ooking for repeated observations before making an interpretation, the atter making an interpretation about a specific observation(5) Time-series anaysis. Specificay ooking at empirica trend(s) over time for a dependent variabe and comparing this empirica trend with one or more theoretica predictions. Like pattern matching, but expicity invoving statistic techniques. Simpe time series might Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 75

102 APPENDIX 3 invove a inear trend for a singe dependent variabe; more compex series might invove non-inear trends and/or mutipe variabes. The researcher must identify the specific indicator(s) to be traced over time, the time intervas to be covered, and the presumed reationships among events prior to coecting the actua data.(1) Logic modes: Describe a repeated cause-and-effect sequence of events inked together (i.e. Provides an initia hypothesis about the case and then provides a framework for anaysing the data. Can use quantitative, quaitative or both kinds of data. The need to consider the infuence of rea-word and other contextua conditions wi vary between studies.(1) Crosscase synthesis. Appies ony to mutipe cases. Synthesising two or more independent cases can be more robust than having just a singe case. Empirica data from mutipe cases coud be used to examine a theory, of be combined statisticay for precision (i.e. meta-anaysis)(1) Essentia Desirabe Not necessary Discuss pausibe riva expanations for the observed data(1) e.g. Nu hypothesis - the observation is soey due to chance (1) Threats to vaidity e.g. poor instrumentation, regression seection(1) Investigator bias e.g. experimenter effect, reactivity in fied research(1) Direct riva e.g. resuts due to intervention B, not intervention A(1) Co-minged riva e.g. intervention A pus one or more other interventions contributed to the resuts(1) Impementation riva - resuts due to the impementation process, rather than the substantive intervention(1) Riva theory - a theory different to the origina theory expains the resuts better Super riva - a force arger than but incuding the intervention accounts for the resuts(1) Societa riva -socia trends, not any particuar force or intervention account for the resuts(1) Essentia Desirabe Not necessary Identify software and describe how it was used(1) Essentia Desirabe Not necessary 76 NIHR Journas Library

103 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Present raw data (incuding iustrative quotes) where necessary(2,5) Essentia Desirabe Not necessary Omit secondary data that is not essentia for understanding and evauating the case study anaysis(4) Essentia Desirabe Not necessary Present data in tabuar form to summarise and compress data(4) Essentia Desirabe Not necessary Array and dispay data in different ways(1) Essentia Desirabe Not necessary Describe how promising patterns, insights and concepts were identified(1) Essentia Desirabe Not necessary Describe the criteria used to maintain the overa quaity of a case study(1, 12) Essentia Desirabe Not necessary Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 77

104 APPENDIX 3 Address the concept of construct vaidity (i.e. identifying correct operationa measures for the concepts being studied)(1) Essentia Desirabe Not necessary Address the concept of interna vaidity [in expanatory or causa studies](i.e. estabishing a causa reationship, whereby certain conditions are beieved to ead to other conditions, as distinguished from spurious reationships)(1,12) Essentia Desirabe Not necessary Address the concept of externa vaidity (i.e. defining the domain to which a study s findings can be generaised)(1) Essentia Desirabe Not necessary Address the concept of reiabiity (i.e. demonstrating that the operations of a study can be repeated with the same resuts)(1) Essentia Desirabe Not necessary Describe how trianguation was carried out,(1) especiay in confirming and disconfirming major assertions(5) e.g. data trianguation (vaidation); (1, 5) investigator trianguation(1, 5); theory trianguation(1, 5;) methodoogica trianguation(1, 5) Essentia Desirabe Not necessary 78 NIHR Journas Library

105 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Outine a chain of evidence that aows the reader to foow the derivation of any evidence from initia research questions to utimate case study concusions, via the coected data(1,4,9,10) Essentia Desirabe Not necessary Do you have any other comments about the data anaysis section? (an opportunity to add more items wi be given ater in this survey) Interpreting the resuts (Section 5 of 7)Pease rate how important it is to incude the foowing items when interpreting and discussing the resuts of an organisationa case study State any caveats about the study(1) Essentia Desirabe Not necessary Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 79

106 APPENDIX 3 Rate the importance of the foowing when describing the strengths and weaknesses of the case study Describe any inherent shortcomings in the design and anaysis and how these might have infuenced the findings(1) Consider the appropriateness of methods used for the question and subject matter and why it was that quaitative methods were appropriate(10) Discuss the samping (or case seection) and expanation of samping strategy(10) Discuss the data anaysis (was it conducted in a systematic way and was it successfu in incorporating a observations and deaing with variation) (10) Discuss the worth & reevance of the research (10) Draw attention to any discrepant data evidence that compicates emerging understanding(7) Discuss the representativeness of data incorporate a shades of opinion(7) 80 NIHR Journas Library

107 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Dispay enough evidence for the reader to reach their own concusions(1, 10) Essentia Desirabe Not necessary Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 81

108 APPENDIX 3 Rate the importance of the foowing for aowing the reader to reach their own concusion Use description to provide the reader with a vicarious experience, or a sense of being there in person, and to enabe understanding of the experience from the informants perspectives.(3) Try to anticipate what vicarious experiences wi do for the reader, and organize the manuscript in a way that faciitates naturaistic generaization(5) Provide enough raw data prior to interpretation for readers to consider their own aternative interpretations(5) Ensure that the assertions are sound, neither over- nor under-interpreting the data (5) Outine the researcher s perspective and reationship to the case(s). The audience needs to understand researcher s roe and perspective to accept findings(5, 13, 14) Ensure the account is refexive i.e. Sensitivity to the ways in which the researcher and research process have shaped the data 82 NIHR Journas Library

109 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 coection and provision of sufficient information of research process for readers to judge(10) Do you have any other comments about the interpretation section? (an opportunity to add more items wi be given ater in this survey) Sharing the resuts and concusions (Section 6 of 7) Pease rate how important it is to incude the foowing items when reporting and disseminating the findings of an organisationa case study Define the audience, whether for written or ora compositions(1) Essentia Desirabe Not necessary Identify the reevant stakehoders(2) Essentia Desirabe Not necessary Identify the researcher position. If the researcher has a cose reationship or a past history with the case being studied, this information shoud be made transparent(3) Essentia Desirabe Not necessary Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 83

110 APPENDIX 3 Be very cear about the research outcomes and how the organization(s) wi benefit from invovement(4) Essentia Desirabe Not necessary Aim for a thoughtfu, baanced, and transparent tone of reporting(1) Essentia Desirabe Not necessary Ensure the report is easy to read(5) Essentia Desirabe Not necessary Aim for a sense of story to the presentation(5) Essentia Desirabe Not necessary Think about narrative dramaturgicay i.e. in terms of actors, roes and stages(6) Essentia Desirabe Not necessary Consider the most appropriate overa reporting structure(1, 3, 4) e.g. Linearanaytic(1); Comparative(1); Chronoogica(1); Theory-buiding(1), Suspense (1); Unsequenced(1); A chronoogica or biographica deveopment of the case(5); A researcher s view of coming to know the case(5); Description one-by-one of severa major components of the case(5) Essentia Desirabe Not necessary 84 NIHR Journas Library

111 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Review and re-compose the report unti done we, using the foowing techniques: Where possibe have informants / participants review the draft report(1) Consut with a range of experts with diverse points of view during after drafting concusions(9) Revise report taking account of feedback from stakehoders(2) Incude the reactions of data sources (and other prospective readers) to the accounts(5) Check ideas and expanations with those in the cuture (e.g. organization)(7) Be refective and have feedback workshops with on site coaborators to road test eary formuations(10) Pubish the report(2) Essentia Desirabe Not necessary Disseminate to scientific (exporatory and expanatory case studies) and poicy audiences (exporatory and exampe case studies)(2) Essentia Desirabe Not necessary Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 85

112 APPENDIX 3 Do you have any other comments about the sharing the resuts section? (an opportunity to add more items wi be given ater in this survey) Further essentia items (Section 7 of 7) Pease add any additiona items that you think are essentia to a set of reporting standards for organisationa case studies. Pease be as concise as possibe; these items wi feed into the second round of the survey.pease separate mutipe items with a semi-coon (;) Describing the design Background, context and theory Describing the data coection Describing the data anaysis Interpreting the resuts Sharing the resuts and concusions Other (not captured by the headings above) If you think that additiona headings are required to capture the essentia items, or that the current headings shoud be reordered, give detais beow (pease be as concise as possibe) Origina items were drawn from the foowing texts: 1. Yin RK. Case study research: design and methods. Thousand Oaks, Caifornia: SAGE Pubications; Huws U, Dahmann S. Quaity standards for case studies in the European Foundation. Dubin: European Foundation for the Improvement of Living and Working Conditions, Moore TS, Lapan SD, Quartaroi MT. Case study research. In: Lapan SD, editor. Quaitative research: an introduction to methods and designs. San Francisco, CA: Jossey- Bass; p Darke P, Shanks G, Broadbent M. Successfuy competing case study research: combining rigour, reevance and pragmatism. Information Systems Journa Oct;8(4): PubMed PMID: WOS: Pubmed Centra PMCID: Incude. Engish. 5. Stake RE. The art of case study research. Thousand Oaks, CA: Sage Pubications; p. 6. Thomas G. How to do your case study : a guide for students and researchers. Los Angees: Sage; Giham B. Case study research methods. London Continuum; Kaarbo J, Beasey RK. A practica guide to the comparative case study method in poitica psychoogy. Poit Psycho Jun;20(2): PubMed PMID: WOS: Pubmed Centra PMCID: Incude. Engish. 9. Greene D, David JL. A research design for generaizing from mutipe case studies. Eva Program Pann. 1984;7: PubMed PMID: Peer Reviewed Journa: Pubmed Centra PMCID: Incude. 10. Fitzgerad L, Dopson S. Comparative case study designs: their utiity and deveopment in organizationa research. In: Buchanan DA, Brynam A, editors. The Sage handbook of organizationa research methods Thousand Oaks, CA: Sage Pubications Ltd; p Crowe S, Cresswe K, Robertson A, Huby G, Avery A, Sheikh A. The 86 NIHR Journas Library

113 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 case study approach. BMC Med Res Methodo. 2011;11:100. PubMed PMID: Pubmed Centra PMCID: Incude. Engish. 12. Meyer CB. A case in case study methodoogy. Fied Methods. 2001;13(4): PubMed PMID: Peer Reviewed Journa: Pubmed Centra PMCID: Incude. 13. Hays PA. Case study research. In: demarrais K, Lapan SD, editors. Foundations for research: methods of inquiry in education and the socia sciences. Mahwah, NJ: Lawrence Erbaum Associates Pubishers; p Gigun JF. A case for case-studies in socia-work research. Soc Work Ju;39(4): PubMed PMID: WOS:A1994NU Pubmed Centra PMCID: Incude. Engish. That is the end of the rating section for this round of the Dephi exercise. A responses are anonymous. In order to assist in ensuring we have an appropriate range and distribution of respondents, we ask you to provide the foowing information in reation to your primary roe/interest: Designation Heath, education, or socia care practitioner Poicy maker Commissioner / funder of research Researcher Research methodoogist Journa editor / board member / invoved in pubishing Other Main area(s) of research interest reated to organisationa case studies How many organisationa case studies have you authored? >10 Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 87

114 APPENDIX 3 How many organisationa case studies have you been invoved with other than as an author? (e.g. peer review; commissioning; advisory roe) >10 What proportion of your work reates to research methodoogy? % 41-60% >60% 88 NIHR Journas Library

115 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Appendix 4 Dephi consensus process round 2 questionnaire Dephi consensus process round 2 questionnaire Note: Direct output from Quatrics survey software OCS dephi round 2 - fina Thank you for your contribution to this work so far. In this second round, you wi have the opportunity to agree or disagree with the responses given in Round One, whether you participated in Round One or not. Foowing from the many hepfu comments we received via the initia questionnaire and others submitted to HS&DR, we woud ike to carify some aspects of this project and survey: Though NIHR HS&DR initiay proposed a common quaity and pubication standard for organisationa case study research, the research team anticipated that generic standards for the conduct of organisationa case studies woud not be feasibe. We therefore chose to focus on quaity of reporting rather than scientific quaity more broady (i.e. Are there aspects of case study reporting that coud faciitate the reading and judgment processes used by peer reviewers and other audiences?) However, if you beieve that a reporting standard is aso not possibe or desirabe, there is now the option to make this cear at the beginning of this round. The initia ist of reporting items were derived from the pubished academic iterature, using the authors own wording wherever possibe. The anguage and paradigmatic assumptions reated to each item are ikey to refect the position of the origina academic author (e.g. the appication of concepts such as vaidity and reiabiity to case study come directy from the pubications of Yin). One aim of this consutation is to estabish whether there can be any consensus on using items from the pubished iterature to inform reporting standards. As researchers, we have expicity tried to avoid making any assumptions or judgements about any of the items. This meant incuding some items that might be considered inappropriate, difficut to understand, not meaningfu, or concerned more with methodoogy than reporting. We anticipated such items to be poory rated in the consutation, and this seems to have broady been the case in Round One. We have no prior view on the ength or content of any future reporting standard that might derive from this work. Just ten of the 112 items from round one met the consensus threshod, and most of these reate to good practice for reporting research in genera. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 89

116 APPENDIX 4 In this round, the items wi again be presented grouped into the foowing sections: Describing the design Background, context and theory Describing the data coection Describing the data anaysis Interpreting the resuts Sharing the resuts and concusions Within each section, you wi be asked to rate two types of item: 1. Items that were initiay rated as Essentia by over 70% of respondents in Round One; 2. Non-essentia items that faied to meet this threshod. These items have been ordered by the ratio of positive to negative responses (i.e. (Essentia+Desirabe)/Not necessary), in decreasing order of positivity. A major issue that was anticipated is the tension between items that can be appied to organisationa case studies in genera, and those that ony appy to a specific paradigm or context. Items may be appropriate to some types of case study and not others. Indeed, this was picked up by the ratings and comments in Round One. In this round, there is the opportunity to distinguish between items that shoud be reported for organisationa case studies in genera, those that shoud be reported for a particuar approach, and those that do not need to be reported. At this stage, the option shoud be used to capture any items that cannot be said to comfortaby satisfy either of the first two options. Therefore, any items you consider to be inappropriate, uninteigibe, irreevant, or unreated to reporting shoud be marked. This second questionnaire shoud take about 30 minutes to compete, and responses shoud be submitted by 5pm (UK time) on Monday 30th March. Did you take part in round 1 of this Dephi exercise? Yes No Do you think that a pubication standard for reporting organisationa case studies is desirabe? Yes No Don't know No opinion Other 90 NIHR Journas Library

117 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Do you think that a meaningfu pubication standard for reporting organisationa case studies is possibe? Yes No Don't know No opinion Other Describing the design: "Essentia" items Over 70% of respondents in round 1 rated the foowing three items as 'essentia' for describing the design of an organisationa case study. Pease state whether you agree that these items shoud be incuded in a generic reporting standard for organisationa case studies. Define the research as a case study (74% rated "Essentia") I agree, this shoud be reported for a organisationa case studies I disagree, this shoud ony be reported for the foowing specific type of organisationa case study: I disagree, this does not need to be reported State the broad aims of the study (84% rated "Essentia") I agree, this shoud be reported for a organisationa case studies I disagree, this shoud ony be reported for the foowing specific type of organisationa case study: I disagree, this does not need to be reported State the research question(s)/hypotheses (79% rated "Essentia") I agree, this shoud be reported for a organisationa case studies I disagree, this shoud ony be reported for the foowing specific type of organisationa case study: I disagree, this does not need to be reported Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 91

118 APPENDIX 4 Describing the design: "Non-essentia" items 24 items for describing the design of an organisationa case study faied to meet 70% consensus in round 1. These items are ranked beow in decreasing order of popuarity (positive/negative rating ratio is given in parentheses). Pease state whether you beieve any of these items shoud be upgraded to become essentia items to be incuded in a generic reporting standard (i.e. "shoud be reported...") for organisationa case studies, or remain excuded from the reporting standard (i.e. ""). Describe why case study is the appropriate method (18.0) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Define the poicy reevance (18.0) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Identify the purpose of the case study (e.g. exporatory, expanatory, evauative, intrinsic, instrumenta) (18.0) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Identify the process(es) (18.0) (e.g. singe or mutipe/coective, embedded/nested, parae, sequentia, retrospective, cross-sectiona, ongitudina) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Identify the specific case(s) and justify the seection (18.0)e.g. Key case (good exampe; cassic or exempary case); Outier case (showing something interesting 92 NIHR Journas Library

119 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 because it is different from the norm); Loca knowedge case (exampe chosen on the basis of persona experience) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe setting/context (physica, economic, historica, cutura, aesthetic) surrounding the case (18.0) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Identify the broad approach(es) (8.5) e.g. Testing a theory; Buiding a theory; Iustrative; Descriptive; Interpretive; Experimenta Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Define the case broady (8.5)e.g. in a case study of neighbouring the case might be defined as either a group of neighbours (peope) or as a geographica neighbourhood (pace) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Show a prior appreciation of the theoretica issues and setting(s) (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 93

120 APPENDIX 4 Acknowedge the potentia conficts between the needs and interests of any sponsoring organizations and the requirements of the research objectives. Show judgment to ensure that an appropriate baance between these is maintained (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: State the impications of the resources avaiabe to the researcher (5.33) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe the boundaries of the case (3.75) i.e. distinguish the subject of the case study (the phenomenon ) from externa data to the case (the context ). Spatia, tempora, and other concrete boundaries shoud be considered. Abstractions (e.g. the concept of neighbouring ) cannot be considered a case. Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe the resoution of etic and emic issues in deveoping the research question. (3.75) (Etic issues are brought in from the researcher from outside; emic issues emerge from inside the case. As the researcher begins to integrate etic and emic, the research question(s) evoves) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: 94 NIHR Journas Library

121 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Describe how the fina research question(s) was deveoped and refined from the broad prima facie question(s) (2.80) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Report "Progressive focusing" i.e. if eary research questions are not heping to thoroughy understand the case, or if new issues become apparent, describe how this changed the research questions (2.80) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Specify the need for recommendations (2.17) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Offer some evidence to the audience that the heterogeneity of the sampe of cases is representative of the heterogeneity of the target popuation (2.17) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Incude "issue questions" or "issue statements" when describing the research question. (2.17) ("Issues" identify one or more aspects of the situation or circumstance surrounding the case, in order to frame the inquiry) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 95

122 APPENDIX 4 Mention any riva cases that were considered (1.71) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: State the deiverabes required (1.71) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Present the case study protoco and describe how it was used (1.11) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe the ikey burden and risks associated with participation for those who (or the site(s) which) comprise the case study (1.11) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe some eary assessments of progress to see if the case shoud be dropped and another seected (1.11) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Background, context, and theory: Non-essentia items A 11 items for describing the background, context and theory of an organisationa case study faied to meet 70% consensus in round 1. These items are ranked beow in decreasing order of popuarity (positive/negative rating ratio is given in parentheses). Pease state whether you beieve any of these items shoud be upgraded to become essentia 96 NIHR Journas Library

123 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 items to be incuded in a generic reporting standard for organisationa case studies (i.e. shoud be reported... ), or remain excuded from the reporting standard (i.e. ). Outine the conceptua structure (i.e. themes or issues) (38.0) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Report the findings of a thorough iterature review (18) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe the theory, propositions and reated issues deveoped to guide the case study and to generaise its findings (18) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Outine the (ogica) connection between the research question(s) and the data coected (8.50) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Define the ogic inking the data to the propositions (i.e. what kind of anaytic techniques were used) (8.50) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: y Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 97

124 APPENDIX 4 Describe any other preparatory research components (e.g. expert interviews, expert workshop) (5.33) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Report whether a piot case study has been conducted (5.33) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Define the criteria for interpreting the findings (3.75) i.e. expicity consider riva expanations (theories) at the outset, to guide decisions about which data shoud be coected, uness using grounded theory Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: For purey exporatory studies without any initia propositions, state a purpose and the criteria by which the exporation is judged successfu or not (3.75) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: State which of the variabes being investigated are hypothesized to be most important for expaining the phenomenon (1.71) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: 98 NIHR Journas Library

125 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Describe whether a range of experts were consuted during the fina stages of deveoping the conceptua framework and report the findings of this consutation (1.38) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describing the data coection: Essentia items Over 70% of respondents in round 1 rated the foowing three items as essentia for describing the coection of data in an organisationa case study. Pease state whether you agree that these items shoud be incuded in a generic reporting standard for organisationa case studies. Describe how data were coected (95% rated "Essentia") I agree, this shoud be reported for a organisationa case studies I disagree, this shoud ony be reported for the foowing specific type of organisationa case study: I disagree, this does not need to be reported Describe the sources of evidence used (95% rated "Essentia") e.g. Documentation; Archiva records; Interviews; Direct observations; Participantobservation; Physica artefacts I agree, this shoud be reported for a organisationa case studies I disagree, this shoud ony be reported for the foowing specific type of organisationa case study: I disagree, this does not need to be reported Describe any ethica considerations and obtainment of reevant approvas, access and permissions (79% rated "Essentia") I agree, this shoud be reported for a organisationa case studies I disagree, this shoud ony be reported for the foowing specific type of organisationa case study: I disagree, this does not need to be reported Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 99

126 APPENDIX 4 Describing the data coection: Non-essentia items 10 items for describing the background, context and theory of an organisationa case study faied to meet 70% consensus in round 1. These items are ranked beow in decreasing order of popuarity (positive/negative rating ratio is given in parentheses). Pease state whether you beieve any of these items shoud be upgraded to become essentia items to be incuded in a generic reporting standard for organisationa case studies (i.e shoud be reported.. ;), or remain excuded from the reporting standard (i.e. Does not need to be reported ;). Describe the data coection too(s) (e.g. questionnaire or observation protoco, incuding a description of any pioting or fied testing of the too) (18) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe the ikey impact of the researcher on events and the behaviour of participants at the case study site, and the researcher's own beiefs, vaues and prior assumptions (18) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe the observation pan and how it was deveoped (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe how the data were coded (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: 100 NIHR Journas Library

127 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Describe whether the data provided an up cose and in-depth coverage of the case (5.33) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Search for data unti saturation is reached, that is, the evidence becomes redundant, with no new information (5.33) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe data protection measures (3.75) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: State whether a comprehensive case study database, in which the raw data can be inspected (incuding notes, documents, tabes and narratives) is avaiabe to readers (2.17) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: List evidence sources in order of importance; give further detais about specific items within each source (1.71) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 101

128 APPENDIX 4 State that a the evidence was examined (1.38) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describing the data anaysis: Essentia items Over 70% of respondents in round 1 rated the foowing item as essentia ; for describing the anaysis of an organisationa case study. Pease state whether you agree that these items shoud be incuded in a generic reporting standard for organisationa case studies. Describe the anaysis methods (90% rated "Essentia") I agree, this shoud be reported for a organisationa case studies I disagree, this shoud ony be reported for the foowing specific type of organisationa case study: I disagree, this does not need to be reported 102 NIHR Journas Library

129 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Describing the data anaysis: Non-essentia items ; 18 items for describing the background, context and theory of an organisationa case study faied to meet 70% consensus in round 1. These items are ranked beow in decreasing order of popuarity (positive/negative rating ratio is given in parentheses). Pease state whether you beieve any of these items shoud be upgraded to become essentia items to be incuded in a generic reporting standard for organisationa case studies (i.e. shoud be reported... ;), or remain excuded from the reporting standard (i.e. ;). State whether an inductive (e.g. grounded) or deductive (e.g. hypothesis testing / theoretica framework) approach to the anaysis has been taken (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Present raw data (incuding iustrative quotes) where necessary (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe how promising patterns, insights and concepts were identified (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Address the concept of externa vaidity (8.5)(i.e. defining the domain to which a study s findings can be generaised) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 103

130 APPENDIX 4 Describe the anaytic approach in detai (5.33) e.g. Pattern matching; Expanation buiding; Time-series anaysis; Logic modes; Cross-case synthesis. Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Identify software and describe how it was used (5.33) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Address the concept of interna vaidity [in expanatory or causa studies] (5.33)(i.e. estabishing a causa reationship, whereby certain conditions are beieved to ead to other conditions, as distinguished from spurious reationships) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe how trianguation was carried out, especiay in confirming and disconfirming major assertions (5.33) e.g. data trianguation (vaidation); investigator trianguation; theory trianguation; methodoogica trianguation Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Outine a chain of evidence that aows the reader to foow the derivation of any evidence from initia research questions to utimate case study concusions, via the coected data (5.33) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: 104 NIHR Journas Library

131 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Discuss pausibe riva expanations for the observed data (3.75) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Describe the criteria used to maintain the overa quaity of a case study (3.75) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: In coective case studies, describe anaysis of data reating to the individua component cases first, before making comparisons across cases (2.80) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Structure the reporting of the anaysis around the research questions (2.17) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Omit secondary data that is not essentia for understanding and evauating the case study anaysis (2.17) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 105

132 APPENDIX 4 Address the concept of construct vaidity (2.17) (i.e. identifying correct operationa measures for the concepts being studied) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Present data in tabuar form to summarise and compress data (1.71) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Address the concept of reiabiity (1.71)(i.e. demonstrating that the operations of a study can be repeated with the same resuts) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Array and dispay data in different ways (1.11) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Interpreting the resuts: Essentia items Over 70% of respondents in round 1 rated the foowing three items as essentia for interpreting the resuts of an organisationa case study. Pease state whether you agree that these items shoud be incuded in a generic reporting standard for organisationa case studies. 106 NIHR Journas Library

133 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Describe any inherent shortcomings in the design and anaysis and how these might have infuenced the findings (79% rated "Essentia") I agree, this shoud be reported for a organisationa case studies I disagree, this shoud ony be reported for the foowing specific type of organisationa case study: I disagree, this does not need to be reported Consider the appropriateness of methods used for the question and subject matter and why it was that quaitative methods were appropriate(79% rated "Essentia") I agree, this shoud be reported for a organisationa case studies I disagree, this shoud ony be reported for the foowing specific type of organisationa case study: I disagree, this does not need to be reported Discuss the data anaysis (i.e. was it conducted in a systematic way and was it successfu in incorporating a observations and deaing with variation) (74% rated "Essentia") I agree, this shoud be reported for a organisationa case studies I disagree, this shoud ony be reported for the foowing specific type of organisationa case study: I disagree, this does not need to be reported Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 107

134 APPENDIX 4 Interpreting the resuts: Non-essentia items 11 items for describing the background, context and theory of an organisationa case study faied to meet 70% consensus in round 1. These items are ranked beow in decreasing order of popuarity (positive/negative rating ratio is given in parentheses). Pease state whether you beieve any of these items shoud be upgraded to become essentia items to be incuded in a generic reporting standard for organisationa case studies (i.e. shoud be reported.. ), or remain excuded from the reporting standard (i.e. ). Draw attention to any discrepant data / evidence that compicates emerging understanding (38) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Ensure that the assertions are sound, neither over- nor under-interpreting the data (38) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: State any caveats about the study (18) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Discuss the samping (or case seection) and expanation of samping strategy (18) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: 108 NIHR Journas Library

135 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Discuss the worth and reevance of the research (18) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Dispay enough evidence for the reader to reach their own concusions (18) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Use description to provide the reader with a "vicarious experience", or a sense of being there in person, and to enabe understanding of the experience from the informants' perspectives. (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Outine the researcher's perspective and reationship to the case(s). The audience needs to understand researcher's roe and perspective to accept findings (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Ensure the account is refexive i.e. Sensitivity to the ways in which the researcher and research process have shaped the data coection and provision of sufficient information of research process for readers to judge (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 109

136 APPENDIX 4 Discuss the representativeness of data - incorporate a shades of opinion (5.33) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Provide enough raw data prior to interpretation for readers to consider their own aternative interpretations (1.71) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Sharing the resuts and concusions: Non-essentia items A 17 items for describing the background, context and theory of an organisationa case study faied to meet 70% consensus in round 1. These items are ranked beow in decreasing order of popuarity (positive/negative rating ratio is given in parentheses). Pease state whether you beieve any of these items shoud be upgraded to become essentia items to be incuded in a generic reporting standard for organisationa case studies (i.e. shoud be reported... ), or remain excuded from the reporting standard (i.e. ). Identify the researcher position. If the researcher has a cose reationship or a past history with the case being studied, this information shoud be made transparent (38) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Pubish the report (18) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: 110 NIHR Journas Library

137 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Disseminate to scientific (exporatory and expanatory case studies) and poicy audiences (exporatory and exampe case studies) (18) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Aim for a thoughtfu, baanced, and transparent tone of reporting (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Ensure the report is easy to read (8.5) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Aim for a sense of story to the presentation (5.33) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Be very cear about the research outcomes and how the organization(s) wi benefit from invovement (3.75) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 111

138 APPENDIX 4 Expicity consider the most appropriate overa reporting structure(3.75) e.g. Linearanaytic; Comparative; Chronoogica; Theory-buiding; Suspense ; Unsequenced; A chronoogica or biographica deveopment of the case; A researcher s view of coming to know the case; Description one-by-one of severa major components of the case Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Acknowedge revision of the report taking account of feedback from stakehoders (3.75) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Define the intended audience (2.80) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Report checking ideas and expanations with those in the cuture (e.g. organization) (2.80) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Identify the reevant stakehoders (2.17) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: 112 NIHR Journas Library

139 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Where possibe have informants / participants review the draft report (2.17) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Be refective and have feedback workshops with on site coaborators to "road test" eary formuations (2.17) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Think about narrative dramaturgicay i.e. in terms of actors, roes and stages (1.38) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Consut with a range of experts with diverse points of view during after drafting concusions (1.38) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Incude the reactions of data sources (and other prospective readers) to the accounts (1.38) Shoud be reported for a organisationa case studies Shoud be reported for the foowing type of organisationa case study: Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 113

140 APPENDIX 4 That is the end of the rating section for this round of the Dephi exercise. A responses are anonymous. In order to assist in ensuring we have an appropriate range and distribution of respondents, we ask you to provide the foowing information in reation to your primary roe/interest: Designation Heath, education, or socia care practitioner Poicy maker Commissioner / funder of research Researcher Research methodoogist Journa editor / board member / invoved in pubishing Other Main area(s) of research interest reated to organisationa case studies How many organisationa case studies have you authored? >10 How many organisationa case studies have you been invoved with other than as an author? (e.g. peer review; commissioning; advisory roe) >10 What proportion of your work reates to research methodoogy? % 41-60% >60% 114 NIHR Journas Library

141 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Appendix 5 Respondent comments from round 1 Do you have any other comments about the design section? (An opportunity to add more items wi be given ater in this survey) Text response I m assuming you re referring to quaitative case studies, but some of the statements above have a very quantitative fee to do them and fee a bit out of pace. Hepfu to know the case study method iterature that are being used as source references, they do not a agree on the key eements of case study design. Burdens and risks are the business of ethics committees, so reporting of ethics approvas may act as a proxy for reporting in the paper. The answer to many of these questions wi depend very substantiay on the design used, e.g. inductive ethnography is very different from a theoreticay based study. Some of the questions asked impy to my mind an overspecification and formaisation of the case study process, e.g. ast one a protoco may evove rather than being fixed at the start of the study. I am sceptica of a attempts to reduce good, refective quaitative research to a set of mandatory steps. I particuary don t ike the insistence on a forma research question (as opposed to a topic/area of interest) which can constrain good exporatory case studies with a broader aim of just understanding what s going on. This is why I am not prepared to tick essentia against many of these things, though they may be good in many cases. My understanding of this Dephi is that it reates to description/presentation of case studies for externa audiences. I have answered it accordingy. However, the items under Rate the importance of the foowing toos and techniques for describing deveopment of the fina research question did not seem to be about reporting, so I strugged sighty with these. There are aso two suggestions in the fina section of this page that I did not understand ( State the deiverabes required and Specify the need for recommendations ). I tried to eave these unanswered but the web page woud not et me, so I have put them down as not necessary but this may be because of my misunderstanding of what they mean. Genera comment you haven t provided the option of saying something ike not appropriate rather than not necessary. This pushes the respondent to answer not necessary when they have some issues with the question. The meaning is not the same. AS I coudn t continue without answering the questions I was not happy with answering, I have ticked desirabe for them. I m reay not sure about the vaidity of a survey where it is not possibe to avoid answering a question that you don t fee is cear or we stated. Q1 hard to answer as some studies may or may not be defined as case studies, depending on how you frame or think about them. This made the question as presented difficut to answer. Q Identifying the purpose boundaries of the case. I think this is essentia but found the question difficut to answer as presented as one might define a case in a more systems- based way, so suggesting the context is externa was not a hepfu way of framing this question, in my view. Heterogeneity of the cases as representative: I coudn t answer this in the terms set. It shoud be essentia to say something about the type of case and whether it can be considered representative or not if it is caiming to be so but as one of your prior questions note, cases may sometimes be seected for quite different reasons than representativeness. State the research questions/hypotheses yes (I have put essentia), but in some studies that are very exporatory, even stating a research question might be considered in appropriate uness constructed broady enough. In the foowing question, I have answered desirabe but fet this was difficut to respond to it is aso poory Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 115

142 APPENDIX 5 framed. Some case studies in their nature woud avoid coming to something fina. It depends how you interpret fina. Aso, I wasn t sure what the question was reay asking. Do you mean that the write up shoud describe the process of refining the questions as part of the study, in reevant studies? Or beforehand? Or both? State the deiverabes required I didn t understand this question. Do you mean by funders or externa agencies? Or, if reevant, the organisation being studied? The possibe responses are very imiting. The authors seem to have worked out what they think is best and are asking do you agree with us? Many of these questions are not inteigibe and seem premised on a very positivist word view. Some kinds of organisationa case study woud be ess dependent on a prior research question/hypothesis but it is a good discipine to keep checking in on the emerging research question/focus during the course of research. For some of these answers, I wanted to answer it depends if case studies were being used in an evauative context, then framing around contros or comparators may be essentia, ess so if more exporatory purpose behind the research. Do you have any other comments about the background, context and theory section? (An opportunity to add more items wi be given ater in this survey) Text response Again, a bit confused by some of these statements how can you know whether exporation was successfu? Aso tak of variabes concerns me very quantitative anguage surey we are searching of understandings and expanations rather than reducing things down to what variabes predict what? some of the words here variabes, hypotheses are very strange in this context indeed. a conceptua framework is ordered around concepts and not facts or events, as wrongy impied in the first statement. The further I get into this the more uncomfortabe I fee about the rigidity of the assumptions underying the questions. It a seems too deterministic, and I am not reassured by a tiny uness using grounded theory get-out cause. Personay I prefer that studies shoud report a detaied iterature review, but I am conscious of the fact that some phiosophies discourage a ot of prior iterature review, instead doing this work as part of the anaysis process foowing ines of enquiry. This then does raise a reporting question of how and where in a report the reevant iterature and theories are discussed. You ve assumed the paradigm is one of variabes. I recommend Ramier and Pentand Management Impications in Information Systems Research: The Untod Story. Journa of the Association for Information Systems Voume 10, Issue 6, pp , June Aso Bent Fyvbjerg Five Misunderstandings About Case-Study Research : Quaitative Inquiry, vo. 12, no. 2, Apri 2006, pp Again I find these questions impossibe to answer without more context. There shoud be a box for a non-response/question uncear. I do not share the stated assumption that reporting standards are easiy or meaningfuy distied into a checkist of standards or even desirabe. There is a need for paradigm differences and theoretica differences which this questionnaire fais to aow. 116 NIHR Journas Library

143 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Do you have any other comments about the data coection section? (An opportunity to add more items wi be given ater in this survey) Text response Do you mean how the data was anaysed? Coding is ony one part of the process of quaitative data anaysis.... Recruitment and criteria for how the study participants were identified, e.g. stakehoder, practitioner with speciaist knowedge and who was excuded for whatever reason, often pragmatic choices have to be made and that needs to be made expicit. Raw data is ikey to be identifiabe, so ethicay it shoud ony be made avaiabe to readers with the consent of participants. The in depth question is very odd, not sure what it means at a. But this is a just characteristics of good quaitative research reporting, not specific to case studies. Again, not a of these seemed to reate to reporting, e.g. Search for data unti saturation is reached, that is, the evidence becomes redundant, with no new information is a methodoogica question, not a question of presentation. See Fyvbjerg s paper. So many of these questions are suited to more nuanced answers. Data saturation is not a concept a quaitative researchers depoy for exampe. I question the inherent assumptions and premise of some of the questions. Do you have any other comments about the data anaysis section? (An opportunity to add more items wi be given ater in this survey) Text response Again very quantitative focused criteria I think its very important to be abe to substantiate any anaytica caims made within a case study, but I woud not phrase this in terms of interna vaidity wrong concept to understand it. Some of the statements above reate, and are therefore more or ess important, to the type of case study and the underying assumptions of the case study approach. Some of these terms or ideas woud be contested by some quaitative researchers as they don t refect the essentia interpretive and emergent nature of good quaitative research. Don t understand what is meant by Describe the criteria used to maintain the overa quaity of a case study. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 117

144 APPENDIX 5 Researchers may use other reevant concepts than those given here, e.g. as per Guba and Lincon s typoogy. There are other possibe approaches apart from inductive or deductive. Some of these are very obvious. But I m not sure you ve covered every eement. And I m not sure this is reay a Dephi. It s more a do you agree with us questionnaire. I think these questions derive from a very positivist understanding and impicit ogic mode. They are mosty not appropriate or meaningfu for those coming from an interpretativist tradition. I think the choice boxes are too narrow and woud ike to register not appropriate rather than not necessary for many answers. Do you have any other comments about the interpretation section? (An opportunity to add more items wi be given ater in this survey) Text response Representativeness is a misnomer here quaitative research does not search for statistica representativeness in the same way that quant research does. You shoud read Nick Emme s book on samping... Reation to theory may aso be key in generating an interpretation. Suddeny a section which makes sense. It s a about the credibiity and refexivity of the construction of the story, not foowing a set of process rues. There is aways an issue of concern over how much raw data to incude given that the data tend to be very detaied and buky. Aso, incusion of arger amounts can be very tricky in such studies when trying to maintain confidentiaity so the baance can be very chaenging. Respondents may sometimes be in a position where simpy disguising name and roe and ceary identifying detais may not be sufficient as the roe is quite specific. There are aso debates about the roe of the researcher and responsibiity to anayse the data with care rather than resort to presenting arge amounts of raw data in the hope that the data wi speak for themseves. This requires a ot of eements, many of which have been referred to in the questions here. One of the above questions impied that the case study is excusivey quaitative. Most good case studies contain some quant data. Again I am unconvinced at trying to produce standards or back and white answers to such highy contextuaised and creative interpretative processes. 118 NIHR Journas Library

145 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Do you have any other comments about the sharing the resuts section? (An opportunity to add more items wi be given ater in this survey) Text response Unsure how many of these statements are case study specific, many woud be true for any research report. These questions are difficut to answer as athough I beieve sharing is very important there are differing views as to how to do it. Sometimes case studies revea uncomfortabe truths. We a ook in the mirror sometimes and fee disappointed or want to see a different image. there may aso be considerabe differences and conficts of perspective between different actors and parties in a case. it is essentia in my view to feedback in some fashion uness there are very particuar barriers to doing this, and to take the responses into fu consideration. This can be very informative and reveaing in itsef, but may not aways be straightforward. This reads as siy Aim for a thoughtfu, baanced, and transparent tone of reporting. Who is going to say aim for a thoughtess, unbaanced and opaque tone. So why ask this? It might be worth considering whether Van Maanen s reaist, impressionist or confessiona genres are most appropriate. Experts are ikey to disagree and hence you might end up with some data worth anaysing. These cosed answers force the respondent into a very narrow set of choices. Most of the reporting categories woud need to conditiona on the type of report, type of funder and purpose of reporting, a very context-specific. Pubication and push to poicy and service depends on the quaity of research and report! Not a given, athough transparency is utimate aim. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 119

146

147 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Appendix 6 Compete respondent comments from round 2 Do you think that a pubication standard for reporting organisationa case studies is desirabe? Text response It depends on the audience or community. Advanced ethnographic case studies targeted at anthropoogy, cutura studies, socioogy or poicy studies are arguaby distinct from HSR or tria research communities. Aso, how do post-structuraist or even narrative case accounts fit with the idea of standards? Standards might constrain creativity and imagination! A depends how it is used. It is one thing to have a standard that acts as a reference point or aspiration; it is another if this is used inappropriatey to enforce standards taht [that] are not universay suitabe for a research that might be subjected to it. Yes but... recognise heterogeneity of case study research. Define the poicy reevance Shoud be reported for the foowing type of organisationa case study Research focusing upon poicy issues. Ones that are poicy reevant. Ones that focus on poicy or poicy-reated issues. If it is a poicy reevant issue. Identify the process(es) (e.g. singe or mutipe/coective, embedded/nested, parae, sequentia, retrospective, cross-sectiona, ongitudina) Shoud be reported for the foowing type of organisationa case study Not sure what this question ( processes ) means. Comparative case studies. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 121

148 APPENDIX 6 Describe setting/context (physica, economic, historica, cutura, aesthetic) surrounding the case Shoud be reported for the foowing type of organisationa case study Studies of atypica organisations (e.g. organisationa innovations, piot schemes, aternative modes of organisation. Yes... and shoud be part of samping frame too. Identify the broad approach(es) (e.g. testing a theory, buiding a theory, iustrative, descriptive, interpretive, experimenta) Shoud be reported for the foowing type of organisationa case study Reaist evauation, quaitative. Might be many of these things more important to carify the purpose of case study research, as above. Show a prior appreciation of the theoretica issues and setting(s) Shoud be reported for the foowing type of organisationa case study Theory-driven ones. Reaist evauation or quaitative. Theoreticay driven ones. Those aiming to appy or test a specific theory. Acknowedge the potentia conficts between the needs and interests of any sponsoring organisations and the requirements of the research objectives. Show judgement to ensure that an appropriate baance between these is maintained Shoud be reported for the foowing type of organisationa case study Where this has a bearing on the findings. If affects decisions about which sites recruited, etc. 122 NIHR Journas Library

149 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 State the impications of the resources avaiabe to the researcher Shoud be reported for the foowing type of organisationa case study Charity-funded evauations these tend to have much ess resources and often resut in quick and dirty evauations. Where resource constraints may have made the findings biased, incompete or otherwise miseading. Describe the boundaries of the case; that is, distinguish the subject of the case study (the phenomenon ) from externa data to the case (the context ). Spatia, tempora, and other concrete boundaries shoud be considered. Abstractions (e.g. the concept of neighbouring ) cannot be considered a case Shoud be reported for the foowing type of organisationa case study Contextua ones. Inter-organisationa studies; studies of reationships between organisation and its environment. Describe the resoution of etic and emic issues in deveoping the research question. (Etic issues are brought in from the researcher from outside; emic issues emerge from inside the case. As the researcher begins to integrate etic and emic, the research question(s) evoves) Shoud be reported for the foowing type of organisationa case study Participatory or action research case studies. Describe how the fina research question(s) was deveoped and refined from the broad prima facie question(s) Shoud be reported for the foowing type of organisationa case study Reaist evauation. Where the research has theoretica or expanatory ambitions. May not be reevant if prior research question/focus is broad. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 123

150 APPENDIX 6 Report Progressive focusing (i.e. if eary research questions are not heping to thoroughy understand the case, or if new issues become apparent, describe how this changed the research questions) Shoud be reported for the foowing type of organisationa case study Reaist evauation or quaitative. Specify the need for recommendations Shoud be reported for the foowing type of organisationa case study If sponsor aows. Sometimes e.g. NIHR you re not aowed to make rec s. Where the purpose is to make recommendations. Ony if poicy type research. Poicy focussed evauations. I woud couch this as drawing out wider impications for the service. Offer some evidence to the audience that the heterogeneity of the sampe of cases is representative of the heterogeneity of the target popuation Shoud be reported for the foowing type of organisationa case study Quantitative, positivist evauations. Ony where representativeness of this nature is being caimed. Those which aim to offer generaised or generaisabe findings. When the study depends on case heterogeneity. Again, being expicit about samping frame is important even if n = 3 or whatever. 124 NIHR Journas Library

151 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Mention any riva cases that were considered Shoud be reported for the foowing type of organisationa case study Where incusion of the riva cases might prima facie appear ikey to affect the resuts. Coud be usefu, not required. State the deiverabes required Shoud be reported for the foowing type of organisationa case study Maybe for poicy-sponsored research. Present the case study protoco and describe how it was used Shoud be reported for the foowing type of organisationa case study Quantitative or positivist case studies quaitative researchers know that case study research is iterative and evoving and that its perfecty acceptabe for the research to be different from the protoco, quants/ positivists have a probem with this! Does this mean fu interview schedues, document anaysis, etc. Describe the ikey burden and risks associated with participation for those who [or the site(s) which] comprise the case study Shoud be reported for the foowing type of organisationa case study Where the burden and risks may have affected data quaity and avaiabiity, hence study findings. Part of access and consent/governance issues. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 125

152 APPENDIX 6 Describe some eary assessments of progress to see if the case shoud be dropped and another seected Shoud be reported for the foowing type of organisationa case study Good idea, if resources aowed. Outine the conceptua structure (i.e. themes or issues) Shoud be reported for the foowing type of organisationa case study Reaist evauation. Conceptuaised ones. Good to have a sense of theoretica frame or conceptua drivers for seection of cases. Report the findings of a thorough iterature review Shoud be reported for the foowing type of organisationa case study If appropriate. Reaist evauation. Describe the theory, propositions and reated issues deveoped to guide the case study and to generaise its findings Shoud be reported for the foowing type of organisationa case study Positivist ones! Reaist evauation. Those which aim to test theories and/or produce generaisabe findings. Where theory is an important aspect of the case study design. 126 NIHR Journas Library

153 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Outine the (ogica) connection between the research question(s) and the data coected Shoud be reported for the foowing type of organisationa case study Ones that use a ogic mode. I don t ike these questions at a. Where the connection is not immediatey obvious. Define the ogic inking the data to the propositions (i.e. what kind of anaytic techniques were used) Shoud be reported for the foowing type of organisationa case study Positivist ones. Describe any other preparatory research components (e.g. expert interviews, expert workshop) Shoud be reported for the foowing type of organisationa case study If it was done, it needs to be reported. Report whether a piot case study has been conducted Shoud be reported for the foowing type of organisationa case study This is siy. If a piot was done, it needs to be reported. Where there are piots. Where new methods of data coection and/or anayses are being deveoped. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 127

154 APPENDIX 6 Define the criteria for interpreting the findings; that is, expicity consider riva expanations (theories) at the outset, to guide decisions about which data shoud be coected, uness using grounded theory Shoud be reported for the foowing type of organisationa case study If you re using criteria for interpreting, then you need to report those criteria. Some of us don t. Reaist evauation. For theory-driven studies. State which of the variabes being investigated are hypothesised to be most important for expaining the phenomenon Shoud be reported for the foowing type of organisationa case study Quantitative case studies qua case studies woud never use the term variabe. Expanatory case studies. Describe any ethica considerations and obtainment of reevant approvas, access and permissions I disagree, this shoud ony be reported for the foowing specific type of organisationa case study NHS-based ones. Describe the data coection too(s) (e.g. questionnaire or observation protoco, incuding a description of any pioting or fied testing of the too) Shoud be reported for the foowing type of organisationa case study One that you want to pubish in a positivist journa. When new or idiosyncratic data coection methods were used. 128 NIHR Journas Library

155 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Describe the ikey impact of the researcher on events and the behaviour of participants at the case study site, and the researcher s own beiefs, vaues and prior assumptions Shoud be reported for the foowing type of organisationa case study Where integra to service issue and data coection. Where it is reasonabe to have expected such impacts. Where this is appropriate given the nature of data coected. Where more participant/action research methods used. Describe the observation pan and how it was deveoped Shoud be reported for the foowing type of organisationa case study Where observation is used as a key method. Ony where observationa techniques were used. New observationa methods; and where the findings depend heaviy on observationa data. Where observation is a data coection method and where this is appropriate given the design/theoretica approach of the study. Describe how the data were coded Shoud be reported for the foowing type of organisationa case study Where quaitative data has been coected. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 129

156 APPENDIX 6 Describe whether the data provided an up-cose and in-depth coverage of the case Shoud be reported for the foowing type of organisationa case study Where reevant. Surey that s for the reader to determine on the basis of the information you give. Quaitative. Search for data unti saturation is reached; that is, the evidence becomes redundant, with no new information Shoud be reported for the foowing type of organisationa case study Quaitative. Those with no other criteria for the competeness of data coection (in terms of the study aims and RQs). Describe data protection measures Shoud be reported for the foowing type of organisationa case study Where protoco dictates, e.g. ethics committee says so or egay required. Those deaing with data whose coection or use raises ethica questions. Part of ethics/governance not necessariy separate. State whether a comprehensive case study database, in which the raw data can be inspected (incuding notes, documents, tabes and narratives) is avaiabe to readers Shoud be reported for the foowing type of organisationa case study If appropriate. Project website is advisabe. 130 NIHR Journas Library

157 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 State that a the evidence was examined Shoud be reported for the foowing type of organisationa case study Where incompete examination woud reasonaby be expected to make the findings incompete, biased or otherwise miseading. Present raw data (incuding iustrative quotes) where necessary Shoud be reported for the foowing type of organisationa case study Where appropriate. You can t set the rues in stone. Where reevant to reporting audience. Quaitative but don t expect a raw data to be presented, but quotes to iustrate the points made are essentia. Where appropriate given the data that has been coected. Describe how promising patterns, insights and concepts were identified Shoud be reported for the foowing type of organisationa case study Where appropriate, and perhaps using iustrative eements but not exhaustivey. Quaitative. Those using inductive methods. Where appropriate given the design of the study. Address the concept of interna vaidity (in expanatory or causa studies) (i.e. estabishing a causa reationship, whereby certain conditions are beieved to ead to other conditions, as distinguished from spurious reationships) Shoud be reported for the foowing type of organisationa case study For expanatory or causa studies. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 131

158 APPENDIX 6 Describe how trianguation was carried out, especiay in confirming and disconfirming major assertions [e.g. data trianguation (vaidation), investigator trianguation, theory trianguation, methodoogica trianguation] Shoud be reported for the foowing type of organisationa case study Where appropriate. Where data trianguation is used. If this is the approach foowed. Those using mutipe kinds of data (interview, documents, observation etc.). This is appropriate for case studies which expicity say they aimed to trianguate as part of the design. Outine a chain of evidence that aows the reader to foow the derivation of any evidence from initia research questions to utimate case study concusions, via the coected data Shoud be reported for the foowing type of organisationa case study In practice, this is hard to do. Describe the criteria used to maintain the overa quaity of a case study Shoud be reported for the foowing type of organisationa case study Remove the word criteria and repace with approach. Not quite sure what this means. Structure the reporting of the anaysis around the research questions Shoud be reported for the foowing type of organisationa case study Ony those which set out with fixed research questions many wi not. Usefu for presentation. 132 NIHR Journas Library

159 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Omit secondary data that is not essentia for understanding and evauating the case study anaysis Shoud be reported for the foowing type of organisationa case study Where secondary data anaysis was a panned component of the origina research design. Address the concept of construct vaidity Shoud be reported for the foowing type of organisationa case study Quantitative. Present data in tabuar form to summarise and compress data Shoud be reported for the foowing type of organisationa case study Where presentationay cearer than text. Address the concept of reiabiity Shoud be reported for the foowing type of organisationa case study Quantitative. Ony where the methods might seem to raise a prima facie objection that the findings are subjective to the author(s). Discuss the samping (or case seection) and expanation of samping strategy Shoud be reported for the foowing type of organisationa case study Studies of heterogeneous popuations of organisations. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 133

160 APPENDIX 6 Use description to provide the reader with a vicarious experience, or a sense of being there in person, and to enabe understanding of the experience from the informants perspectives Shoud be reported for the foowing type of organisationa case study Depends on reporting format. Outine the researcher s perspective and reationship to the case(s). The audience needs to understand researcher s roe and perspective to accept findings Shoud be reported for the foowing type of organisationa case study Quaitative. Where there might be the appearance of a confict of interest, or the author(s) have a partisan reputation. Where there are participant researchers. Provide enough raw data prior to interpretation for readers to consider their own aternative interpretations Shoud be reported for the foowing type of organisationa case study Where appropriate. Where the findings rest on the baance of compex and ambivaent data. Identify the researcher position. If the researcher has a cose reationship or a past history with the case being studied, this information shoud be made transparent Shoud be reported for the foowing type of organisationa case study Ony where the researcher has a reationship with the case being studied. 134 NIHR Journas Library

161 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Acknowedge revision of the report taking account of feedback from stakehoders Shoud be reported for the foowing type of organisationa case study Where appropriate. Participatory/action research. Define the intended audience Shoud be reported for the foowing type of organisationa case study This is the key factor that shapes most of the responses. Report checking ideas and expanations with those in the cuture Shoud be reported for the foowing type of organisationa case study Where appropriate. Action research/participatory. Where the author(s) have heaviy interpreted or re-structured the origina data. Where possibe have informants/participants review the draft report Shoud be reported for the foowing type of organisationa case study Where appropriate. Study sites shoud usuay have opportunity to review some form of findings. Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 135

162 APPENDIX 6 Be refective and have feedback workshops with onsite coaborators to road test eary formuations Shoud be reported for the foowing type of organisationa case study Where reevant. Participatory/action research. Consut with a range of experts with diverse points of view during after drafting concusions Shoud be reported for the foowing type of organisationa case study Where appropriate. Participatory/action research. Incude the reactions of data sources (and other prospective readers) to the accounts Shoud be reported for the foowing type of organisationa case study Where the author(s) have heaviy interpreted or re-structured the origina data. 136 NIHR Journas Library

163 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Appendix 7 Items with 70% consensus in round 2 Percentage of tota responses Item Shoud be reported for a organisationa case studies Shoud be reported for a specific type of organisationa case study Does not need to be reported Describe the resoution of etic and emic issues in deveoping the research question Describe how the fina research question(s) was deveoped and refined from the broad prima facie question(s) Present the case study protoco and describe how it was used Describe the ikey burden and risks associated with participation for those who [or the site(s) which] comprise the case study Describe any other preparatory research components (e.g. expert interviews, expert workshop) For purey exporatory studies without any initia propositions, state a purpose and the criteria by which the exporation is judged successfu or not State which of the variabes being investigated are hypothesised to be most important for expaining the phenomenon Structure the reporting of the anaysis around the research questions Address the concept of reiabiity (i.e. demonstrating that the operations of a study can be repeated with the same resuts) Use description to provide the reader with a vicarious experience, or a sense of being there in person, and to enabe understanding of the experience from the informants perspectives Provide enough raw data prior to interpretation for readers to consider their own aternative interpretations Pubish the report Disseminate to scientific (exporatory and expanatory case studies) and poicy audiences (exporatory and exampe case studies) Acknowedge revision of the report taking account of feedback from stakehoders Mention any riva cases that were considered State the deiverabes required Describe whether a range of experts were consuted during the fina stages of deveoping the conceptua framework and report the findings of this consutation Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 137

164 APPENDIX 7 Percentage of tota responses Item Shoud be reported for a organisationa case studies Shoud be reported for a specific type of organisationa case study Does not need to be reported Search for data unti saturation is reached; that is, the evidence becomes redundant, with no new information State whether a comprehensive case study database, in which the raw data can be inspected (incuding notes, documents, tabes and narratives) is avaiabe to readers Address the concept of construct vaidity (i.e. identifying correct operationa measures for the concepts being studied) Be very cear about the research outcomes and how the organisation(s) wi benefit from invovement Define the intended audience Where possibe, have informants/participants review the draft/report Be refective and have feedback workshops with onsite coaborators to road test eary formuations Consut with a range of experts with diverse points of view during and/or after drafting concusions Describe some eary assessments of progress to see if the case shoud be dropped and another seected State that a the evidence was examined Omit secondary data that is not essentia for understanding and evauating the case study anaysis Present data in tabuar form to summarise and compress data Array and dispay data in different ways Expicity consider the most appropriate overa reporting structure (e.g. inear-anaytic; comparative; chronoogica; theory buiding; suspense ; unsequenced; a chronoogica or biographica deveopment of the case; a researcher s view of coming to know the case; description one-by-one of severa major components of the case) Identify the reevant stakehoders Think about narrative dramaturgicay, i.e. in terms of actors, roes and stages Incude the reactions of data sources (and other prospective readers) to the accounts Incude issue questions or issue statements when describing the research question. ( Issues identify one or more aspects of the situation or circumstance surrounding the case, in order to frame the inquiry) List evidence sources in order of importance; give further detais about specific items within each source NIHR Journas Library

165 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Appendix 8 Items with no overa consensus Percentage of tota responses Item Shoud be reported for a organisationa case studies Shoud be reported for a specific type of organisationa case study Does not need to be reported State whether an inductive (e.g. grounded) or deductive (e.g. hypothesis testing/theoretica framework) approach to the anaysis has been taken Discuss the samping (or case seection) and expanation of samping strategy Identify the purpose of the case study (e.g. exporatory, expanatory, evauative, intrinsic, instrumenta) Describe the data coection too(s) (e.g. questionnaire or observation protoco, incuding a description of any pioting or fied testing of the too) Draw attention to any discrepant data/evidence that compicates emerging understanding Identify the process(es) (e.g. singe or mutipe/ coective, embedded/nested, parae, sequentia, retrospective, cross-sectiona, ongitudina) Describe setting/context (physica, economic, historica, cutura, aesthetic) surrounding the case Describe how the data were coded Outine a chain of evidence that aows the reader to foow the derivation of any evidence from initia research questions to utimate case study concusions, via the coected data Discuss pausibe riva expanations for the observed data Dispay enough evidence for the reader to reach their own concusions Identify the researcher position. If the researcher has a cose reationship or a past history with the case being studied, this information shoud be made transparent Describe why case study is the appropriate method Define the case broady [e.g. in a case study of neighbouring the case might be defined as either a group of neighbours (peope) or as a geographica neighbourhood (pace)] Aim for a sense of story to the presentation Acknowedge the potentia conficts between the needs and interests of any sponsoring organisations and the requirements of the research objectives. Show judgement to ensure that an appropriate baance between these is maintained Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 139

166 APPENDIX 8 Percentage of tota responses Item Shoud be reported for a organisationa case studies Shoud be reported for a specific type of organisationa case study Does not need to be reported Outine the (ogica) connection between the research question(s) and the data coected Define the ogic inking the data to the propositions (i.e. what kind of anaytic techniques were used) Describe how promising patterns, insights and concepts were identified Address the concept of externa vaidity (i.e. defining the domain to which a study s findings can be generaised) Describe the anaytic approach in detai (e.g. pattern matching, expanation buiding, time-series anaysis, ogic modes and cross-case synthesis) Discuss the worth and reevance of the research Ensure the account is refexive (i.e. Sensitivity to the ways in which the researcher and research process have shaped the data coection and provision of sufficient information of research process for readers to judge) Aim for a thoughtfu, baanced and transparent tone of reporting Ensure the report is easy to read Identify the broad approach(es) (e.g. testing a theory, buiding a theory, iustrative, descriptive, interpretive, experimenta) Report Progressive focusing, i.e. if eary research questions are not heping to thoroughy understand the case, or if new issues become apparent, describe how this changed the research questions Outine the conceptua structure (i.e. themes or issues) Describe the ikey impact of the researcher on events and the behaviour of participants at the case study site, and the researcher s own beiefs, vaues and prior assumptions Present raw data (incuding iustrative quotes) where necessary Identify software and describe how it was used Address the concept of interna vaidity (in expanatory or causa studies) (i.e. estabishing a causa reationship, whereby certain conditions are beieved to ead to other conditions, as distinguished from spurious reationships) Describe the criteria used to maintain the overa quaity of a case study NIHR Journas Library

167 DOI: /hsdr04010 HEALTH SERVICES AND DELIVERY RESEARCH 2016 VOL. 4 NO. 1 Percentage of tota responses Item Shoud be reported for a organisationa case studies Shoud be reported for a specific type of organisationa case study Does not need to be reported Outine the researcher s perspective and reationship to the case(s). The audience needs to understand researcher s roe and perspective to accept findings Show a prior appreciation of the theoretica issues and setting(s) Describe the boundaries of the case; that is, distinguish the subject of the case study (the phenomenon ) from externa data to the case (the context ). Spatia, tempora and other concrete boundaries shoud be considered. Abstractions (e.g. the concept of neighbouring ) cannot be considered a case Describe the theory, propositions and reated issues deveoped to guide the case study and to generaise its findings Describe the observation pan and how it was deveoped Describe data protection measures Define the poicy reevance State the impications of the resources avaiabe to the researcher Report whether a piot case study has been conducted Define the criteria for interpreting the findings; that is, expicity consider riva expanations (theories) at the outset, to guide decisions about which data shoud be coected, uness using grounded theory Describe how trianguation was carried out, especiay in confirming and disconfirming major assertions, for exampe data trianguation (vaidation); investigator trianguation; theory trianguation; methodoogica trianguation In coective case studies, describe anaysis of data reating to/the individua component cases... Discuss the representativeness of data incorporate a shades of opinion Specify the need for recommendations Report the findings of a thorough iterature review Describe whether the data provided an up-cose and in-depth coverage of the case Report checking ideas and expanations with those in the cuture (e.g. organisation) Offer some evidence to the audience that the heterogeneity of the sampe of cases is representative of the heterogeneity of the target popuation Queen s Printer and Controer of HMSO This work was produced by Rodgers et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 141

168

169

170 EME HS&DR HTA PGfAR PHR Part of the NIHR Journas Library This report presents independent research funded by the Nationa Institute for Heath Research (NIHR). The views expressed are those of the author(s) and not necessariy those of the NHS, the NIHR or the Department of Heath Pubished by the NIHR Journas Library