PEER REVIEW HISTORY ARTICLE DETAILS TITLE (PROVISIONAL)

Transcription

1 PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form (see an example) and are provided with free text boxes to elaborate on their assessment. These free text comments are reproduced below. Some articles will have been accepted based in part or entirely on reviews undertaken for other BMJ Group journals. These will be reproduced where possible. ARTICLE DETAILS TITLE (PROVISIONAL) AUTHORS Assessing the efficacy of the electronic patient record system EDeR: Implementation study - Study protocol Bachmann, Lucas; Job, Oliver; Schmid, Martin; Thiel, Michael; Ivic, Sandra VERSION 1 - REVIEW REVIEWER REVIEW RETURNED MILopez Galvez IOBA University of Valladolid, Valladolid,Spain 16-Dec-2012 RESULTS & CONCLUSIONS The article only includes the protocol not the results. It is not enough GENERAL COMMENTS I suggest to publish it with the results with thw analysis included There is no conlcusion The article is incomplete REVIEWER REVIEW RETURNED Frank Larkin, Consultant ophthalmologist, Moorfields Eye Hospital, London, UK. I have no competing interests. 31-Dec-2012 THE STUDY GENERAL COMMENTS 1. It may be inappropriate to evaluate clinician performance using the electronic record before any necessary learning time using the software has elapsed. I realise that the planned second analysis at one year would probably identify any such effect due to inexperience. 2. Refs #3-5, 8, 9 are incomplete (year of publication). This manuscript describes the design of a prospective simulated study to examine the factors associated with improved performance using an electronic clinical record versus paper records. They term this a randomised vignette study. The design is based on specific clinical decision points in consultations in a range of ophthalmology sub-specialties. While ophthalmology is itself a small specialty, it does lend itself to electronic patient record keeping better than many others for a number of reasons, including the high proportion of images which are used as diagrams or digital photographs in records (for example fundoscopy findings). My opinion is that the findings in this type of study would be of broad interest in clinical medicine more generally.

2 Specific comments. 1. Having indicated my support for this study, it would be of much higher value if the results were included. I encourage the authors to add their findings and discuss these. The manuscript is short as submitted. 2. As the authors indicate, there has been little research previously published in this field. More critical analysis of those previous reports which they cite would improve the manuscript. 2. Methods & design, P9 line 7: Is it valid to assume standard distribution of any differences in responses to matched vignette pairs? 3. Time of data collection, P9 line 22: If the authors have not set out to examine for any clinician learning effect with software, why not assess competence with software before the first comparative analysis? REVIEWER REVIEW RETURNED Prof. Dr. med. Jürgen Stausberg Ludwig-Maximilians-Universität München Institut für Medizinische Informationsverarbeitung, Biometrie und Epidemiologie (IBE) Germany 16-Jan-2013 GENERAL COMMENTS The authors present a well-thought-out study protocol to assess the electronic patient record system EDeR to be implemented in their ophthalmological department. Main focus of this study is to explore differences in speed and accuracy when searching clinical information using the paper based patient record or the EDeR. The authors also talk about patient s safety, which should be improved by an electronic medical record (EMR), and the possible barriers for a successful use of the EMR. As far as the reviewer understands, the plan is as follows. Patient vignettes are the observational units. The study measures the time needed to solve tasks related to those vignettes. The reviewer assumes that tasks and vignettes are related 1 to 1. Otherwise, if a patient vignette is used for several tasks, the tasks become observational unit. Time is the outcome variable. Intervention is the EMR that is compared with the paper-based patient record. Therefore, the study implements a two group design. The authors assume that the intervention saves 60 seconds time in the mean with a standard deviation of 45 seconds. Instead of randomly assigning vignettes to the groups, the study implements a crossover design in randomly assigning the sequence (EMR first, paper second vs. paper first EMR second) to vignettes. That design is underpinned by a formal correct sample size calculation leading to 7 observational units needed. To make it more complex, the authors include subgroups as well as several observers for each measurement. They include 5 subgroups for clinical specialties related to the patient vignettes and 2 subgroups of level of expertise related to the observers. At least, they will have 2 x 5 x 5 (five tasks each clinical subspecialty) x 2 (EMR and paper) = 500 measurements. To address less reliability having only one observer

3 in each subgroup (experts and beginners) the authors intend to recruit more than the two for the study. To make it more serious, the vignettes are slightly modified according to the group. So if an observer checks the same vignette a second time (e. g. in the EMR), he must read the content again, because the answer is different to the first time (e. g. in the paper-based version). The material for the paper-based-group comes from real patients records; the material for the EMR-group is entered into the EMR specifically for the study. This long description makes clear that the proposed study protocol characterizes a complex study design. Unfortunately, the complexity is not well addressed in the text. Furthermore, it is unclear whether the design is really appropriate to answer the research question. Therefore I recommend 1) to improve the readability of the text by clarifying the points made above on the one hand and 2) to rethink the study design according to the research question on the other hand. It would be helpful if the authors refer 3) to an established schema for study design rather than to describe the details in own words. However, the study idea of the authors in assessing the efficacy of EDeR is essential and a current and important topic that must be discussed. It offers a good basis to explore other essential usability and efficacy questions regarding the use of EMRs. General essential recommendations 4) As long as the time needed to fulfill the tasks is the only measurement, the study is solely related to efficacy as the authors denote time saving. Then, the study does not measure accuracy. Consequently, accuracy should be deleted a study goal. 5) A figure should be added to explain the several entities in the study (vignettes, tasks, sub-specialties, user groups). 6) A major concern with the study design is a potential bias introduced by the unblinded recording of the vignettes into the EMR. The reviewer assumes that the result is an optimized representation of the vignettes in the EMR. This optimized representation will be different from the information available in the EMR in routine use. Comparing the optimized representation of the vignettes in the EMR with an ordinary representation in a paper-based-record is unfair. Addressing this potential bias in the study design is mandatory. Specific essential recommendations 7) The authors should clarify whether the participants of the study will be trained or not in the implemented EDeR prior to the assessment.

4 8) Study population: How big the study population is or will be? For instance, how many ophthalmologists are found in the eye clinic of the cantonal hospital Lucerne? How many junior and senior ophthalmologists are there? How many ophthalmologists already voluntarily agreed to participate in the study? What is the expectation of the recruitment rate? Experimental design: 9) For each out of five different ophthalmologic sub-specialties, the authors will create five tasks based on a patient example. Did some literature lead the choice of these five task types? Or how did the authors come to the definition of the task types? What motivated this choice? The authors should specify if these five tasks are the main consultation s tasks in their ophthalmology department or if these tasks are generally error-prone when using the paper-based form. 10) Each physician from each division has to solve all of the tasks. So, a physician from a specific division will have to solve <5 or 25?> tasks using paper-based records and <5 or 25?> tasks using the EDeR in a randomized sequence. Please specify if the physicians will have to solve all <10 or 50?> tasks at once, that is without break. 11) The evaluation will take place at two time points; immediately after implementation of EDeR and one year after. Please specify if the physicians are already trained in EDeR at the first time point. What about the second time point (one year after)? Also specify if the physicians will be using EDeR during that year or would you keep training them to prepare them to the second evaluation? Detailed research plan: 12) At the assessment/the role of the assessor. Let s suppose a technical problem occurs after a physician has already performed 8 tasks. This problem unfortunately forces the break of the assessment. Shall the physician start all tasks from the beginning at another time point or may he perform only the last tasks later? Sample size calculation: 13) Please check, if the sample size estimation holds true in view of the several subgroups as well as the issue of multiple testing introduced with the subgroups. 14) Having 20 minutes time for a whole consultation, 60 seconds time reduction with a single task seems unrealistic. Please clarify how the total time reduction for a whole consultation is estimated by observing small tasks. Discussion: We presume that the implementation of the electronic medical record software EDeR will have a positive impact on the efficiency of

5 the doctors, which will result in an increase of consultations per day. 15) Good and fine, but what is more important: the increase of consultations per day (through decrease of consultation time: 1 min gained) or the improvement of patients healthcare (improved quality of consultation, patient safety)? Further recommendations 16) Abstract Introduction: replace the second sentence with Critics, however, claim that they are hard to read, because of illegible handwriting, and uncomfortable to use. 17) Abstract Methods and Analysis: there is an 'e' too much in the first sentence. Background: second paragraph, last sentence: As a result of these problems should be As a solution to these problems. 18) Background: third paragraph, last sentence: the second word 'understanding' should be 'understand'. 19) Last paragraph, first sentence: there is an 'e' too much in the first sentence (In this protocol we describe e a study exploring ). 20) Methods/Design Study population, first sentence: change in in into. 21) Methods/Design Data collection Point of time of data collection: Second part of the second sentence should be rewritten. Did you mean: with the help of a medical record? 22) Even if there is just one table provided in the paper, the authors should give it a number when referencing to it. 23) References related to the evaluation of medical informatics interventiones is misssing. 24) The selection of references according to the research question is arbitrary. 25) It is unclear whether a conflict of interest exists due to the involvement of the authors in the development of the EMR-system. Please include a statement related to the involvement of the authors.

6 VERSION 1 AUTHOR RESPONSE # Reviewer: M.I. Lopez Galvez Our reply 1 The article only includes the protocol not the results. It is not enough Thank you. This is the protocol for a study that will be still executed. BMJ open offers publication of protocols and we think that this is very useful to share and discuss methods of studies before their execution. 2 I suggest to publish it with the results with the analysis included There is no conclusion Reviewer: Frank Larkin, Please see our reply to query 1 Our reply 1 It may be inappropriate to evaluate clinician performance using the electronic record before any necessary learning time using the software has elapsed. I realise that the planned second analysis at one year would probably identify any such effect due to inexperience. Thank you for the favorable assessment of our proposal. 2 Refs #3-5, 8, 9 are incomplete (year of publication). Thank you the references have been revised. 3 This manuscript describes the design of a prospective simulated study to examine the factors associated with improved performance using an electronic clinical record versus paper records. They term this a randomised vignette study. The design is based on specific clinical decision points in consultations in a range of ophthalmology subspecialties. While ophthalmology is itself a small specialty, it does lend itself to electronic patient record keeping better than many others for a number of reasons, including the high proportion of images which are used as diagrams or digital photographs in records (for example fundoscopy findings). My opinion is that the findings in this type of study would be of broad interest in clinical medicine more generally. 4 Having indicated my support for this study, it would be of much higher value if the results were included. I encourage the authors to add their findings and discuss these. The manuscript is short as submitted. 5 As the authors indicate, there has been little research previously published in this field. More critical analysis of those previous reports which they cite would improve the manuscript. Thank you! No reply required. Thank you. We would like to refer to our reply to query number 1 of Dr. Lopez Galvez Thank you no reply required.

7 6 Methods & design, P9 line 7: Is it valid to assume standard distribution of any differences in responses to matched vignette pairs? 7 Time of data collection, P9 line 22: If the authors have not set out to examine for any clinician learning effect with software, why not assess competence with software before the first comparative analysis? Thank you. We checked again with our statistician and he confirmed the adequacy of our approach. Thank you for this important query. We fully agree and clarified this point in the revised paper. (p.8, 1 st para) Prior to the examination baseline data will be collected of each participant.

8 Reviewer Jürgen Stausberg Our reply 1 The authors present a well-thought-out study protocol to assess the electronic patient record system EDeR to be implemented in their ophthalmological department. Main focus of this study is to explore differences in speed and accuracy when searching clinical information using the paper based patient record or the EDeR. The authors also talk about patient s safety, which should be improved by an electronic medical record (EMR), and the possible barriers for a successful use of the EMR. 2 As far as the reviewer understands, the plan is as follows. Patient vignettes are the observational units. The study measures the time needed to solve tasks related to those vignettes. The reviewer assumes that tasks and vignettes are related 1 to 1. Otherwise, if a patient vignette is used for several tasks, the tasks become observational unit. Time is the outcome variable. Intervention is the EMR that is compared with the paper-based patient record. Therefore, the study implements a two group design. The authors assume that the intervention saves 60 seconds time in the mean with a standard deviation of 45 seconds. Instead of randomly assigning vignettes to the groups, the study implements a crossover design in randomly assigning the sequence (EMR first, paper second vs. paper first EMR second) to vignettes. That design is underpinned by a formal correct sample size calculation leading to 7 observational units needed. To make it more complex, the authors include subgroups as well as several observers for each measurement. They include 5 subgroups for clinical specialties related to the patient vignettes and 2 subgroups of level of expertise related to the observers. At least, they will have 2 x 5 x 5 (five tasks each clinical subspecialty) x 2 (EMR and paper) = 500 measurements. To address less reliability having only one observer in each subgroup (experts and beginners) the authors intend to recruit more than the two for the study. To make it more serious, the vignettes are slightly modified according to the group. So if an observer checks the same vignette a second time (e. g. in the EMR), he must read the content again, because the answer is different to the first time (e. g. in the paper-based version). The material for the paper-based-group comes from real patients records; the material for the EMR-group is entered into the EMR specifically for the study. Thank you for the favorable assessment of our protocol. No reply required. This is an excellent analysis of the proposed method. Thank you no reply required.

9 3 This long description makes clear that the proposed study protocol characterizes a complex study design. Unfortunately, the complexity is not well addressed in the text. Furthermore, it is unclear whether the design is really appropriate to answer the research question. Therefore I recommend to improve the readability of the text by clarifying the points made above on the one hand and to rethink the study design according to the research question on the other hand. 4 It would be helpful if the authors refer to an established schema for study design rather than to describe the details in own words. 5 However, the study idea of the authors in assessing the efficacy of EDeR is essential and a current and important topic that must be discussed. It offers a good basis to explore other essential usability and efficacy questions regarding the use of EMRs. Thank you for this query we agree that the description was too brief. We amended the protocol as follows lending substantially from Prof Stausberg s excellent summary in point 2: (p 8, last. para.) Design Matrix Summary This is a two group (EDeR vs. paper based) cross-over (randomly assigning the sequence electronic vs. paper-based per task) study. Time (primary endpoint) and accuracy (secondary endpoint) are the two outcome variables. Intervention is the EDeR that is compared with the paper-based patient record. We assess tasks within 5 clinical subgroups (without examining subgroup effects) and perform the experiment in two groups with different ophthalmologic expertise (junior doctors vs. senior doctors). Each assessor will complete 50 tasks (i.e. 25 tasks based on the electronic patient record and 25 based on the paper patient record). Electronic and paper-based vignettes are slightly different for each task. If an assessor checks the same vignette a second time (e. g. in the electronic version), he or she must read the content again, because the answer is different to the first time (e. g. in the paper-based version). The material for the paperbased-group comes from real patients records and the tasks were selected based on clinical relevance. Thank you we are unaware of a similar application and would be grateful, if you could indicate a paper covering the same topic and applying a similar method. Thank you, no reply required.

10 6 As long as the time needed to fulfill the tasks is the only measurement, the study is solely related to efficacy as the authors denote time saving. Then, the study does not measure accuracy. Consequently, accuracy should be deleted a study goal. 7 A figure should be added to explain the several entities in the study (vignettes, tasks, subspecialties, user groups). 8 A major concern with the study design is a potential bias introduced by the unblinded recording of the vignettes into the EMR. The reviewer assumes that the result is an optimized representation of the vignettes in the EMR. This optimized representation will be different from the information available in the EMR in routine use. Comparing the optimized representation of the vignettes in the EMR with an ordinary representation in a paper-based-record is unfair. Addressing this potential bias in the study design is mandatory. 9 The authors should clarify whether the participants of the study will be trained or not in the implemented EDeR prior to the assessment. We fully agree that the protocol did not clearly specify that accuracy was a secondary outcome of this study. We amended the protocol as follows. (p para.) Primary outcome Gain in speed, when performing the task using the EDeR system Secondary outcome Accuracy of replies given using the EDeR system vs. the paper record We agree with the Reviewer and added a figure to the amended manuscript. Thank you. The Reviewer raises an important issue. Data managers at the Hospital will enter the complete available patient record into EDeR. Thus, the content will be identical in both exposures. We agree that this aspect was not clearly stated in the initial submission. We therefore revised the paper as follows: (p. 7, 2. para) The patient model will come from original paper forms to assure that cases are comparable to the real life situation. The full patient record, containing a slight modification for the contrast examined in a specific task, will be transferred from paper into the electronic patient record system. Thank you. We fully agree and amended the protocol as follows: (p.6, 5 para) Ophthalmologist of the cantonal hospital Lucerne eye clinic who voluntarily agreed to participate in the study will be included. Each participant will receive a general introduction on the usage of the EDeR of half an hour prior to the assessment.

11 10 Study population: How big the study population is or will be? For instance, how many ophthalmologists are found in the eye clinic of the cantonal hospital Lucerne? How many junior and senior ophthalmologists are there? How many ophthalmologists already voluntarily agreed to participate in the study? What is the expectation of the recruitment rate? 11 For each out of five different ophthalmologic subspecialties, the authors will create five tasks based on a patient example. Did some literature lead the choice of these five task types? Or how did the authors come to the definition of the task types? What motivated this choice? The authors should specify if these five tasks are the main consultation s tasks in their ophthalmology department or if these tasks are generally errorprone when using the paper-based form. 12 Each physician from each division has to solve all of the tasks. So, a physician from a specific division will have to solve <5 or 25?> tasks using paper-based records and <5 or 25?> tasks using the EDeR in a randomized sequence. Please specify if the physicians will have to solve all <10 or 50?> tasks at once, that is without break. Thank you for this query. We provide some information in the revised paper. (p.6. 2 nd para) The study is planned to take place at the cantonal hospital of Lucerne in the clinic of ophthalmology. The Eye clinic is run by six consultant ophthalmologists, 12 senior physicians, and 14 junior doctors. And (p. 9, 3 rd para) At the time of writing this protocol 10 senior physicians and 9 junior doctors were willing to participate in the study. Thank you again for this important query. We specified the selection of topics in the amended manuscript as follows: (p.7, 1 st para) The tasks are chosen on the basis of the relevance in clinical practice. Thank you. We clarified this as follows: (p.8, 3 rd para) We will design a reply form into which each participant enters his or her response to the task. This reply form will be handed over to the assessor after completion. Time will be taken by the assessor after completing each entry into this form. In total, each participant will go through 25 pairs of tasks.

12 13 The evaluation will take place at two time points; immediately after implementation of EDeR and one year after. Please specify if the physicians are already trained in EDeR at the first time point. What about the second time point (one year after)? Also specify if the physicians will be using EDeR during that year or would you keep training them to prepare them to the second evaluation? 14 At the assessment/the role of the assessor. Let s suppose a technical problem occurs after a physician has already performed 8 tasks. This problem unfortunately forces the break of the assessment. Shall the physician start all tasks from the beginning at another time point or may he perform only the last tasks later? 15 Please check, if the sample size estimation holds true in view of the several subgroups as well as the issue of multiple testing introduced with the subgroups. Please see our reply to your query 9. Before the first exposure, only a general introduction on the usage of EDeR will be provided that will last about 30 minutes. The training will cover aspects of data collection and saving very briefly. The training will be such, that the basic functions are known. We amended the paper as follows. (p. 6, 3 rd para.) The training will cover aspects of data collection and saving very briefly. The training will be such, that the basic functions of the EDeR are known. On page 10, 3 rd para we also revised as follows: At the second time point, one year after the first assessment, the experiment will be repeated using the same protocol. In the interval between the two measurements, all physicians will be mainly working with the EDeR system. Thank you very much for this important query. We fully agree and added this statement on p.9 1 st para: In case of a technical problem with the electronic system, the assessment will be suspended and the remaining tasks will be completed after restoration of the system. We will only make statistical comparisons on the main effect. Subgroup analyses will be performed in exploratory analyses. Any subgroup result will be marked in the paper as such and the issue of power will be discussed.

13 16 Having 20 minutes time for a whole consultation, 60 seconds time reduction with a single task seems unrealistic. Please clarify how the total time reduction for a whole consultation is estimated by observing small tasks. 17 We presume that the implementation of the electronic medical record software EDeR will have a positive impact on the efficiency of the doctors, which will result in an increase of consultations per day. Good and fine, but what is more important: the increase of consultations per day (through decrease of consultation time: 1 min gained) or the improvement of patients healthcare (improved quality of consultation, patient safety)? 18 Abstract Introduction: replace the second sentence with Critics, however, claim that they are hard to read, because of illegible handwriting, and uncomfortable to use. 19 Abstract Methods and Analysis: there is an 'e' too much in the first sentence. 20 Background: second paragraph, last sentence: As a result of these problems should be As a solution to these problems. 21 Background: third paragraph, last sentence: the second word 'understanding' should be 'understand'. 22 Last paragraph, first sentence: there is an 'e' too much in the first sentence (In this protocol we describe e a study exploring ). 23 Methods/Design Study population, first sentence: change in in into. 24 Methods/Design Data collection Point of time of data collection: Second part of the second sentence should be rewritten. Did you mean: with the help of a medical record? The reviewer raises an interesting question. We performed a small survey among the senior staff at our institution and there was a consensus that a meaningful difference should be at least 1 Minute / Patient. Therefore, this delta was chosen. We do agree that the advantage in terms of speed will not necessarily come from one single task in clinical practice. But still this could be the case for example when checking the number of interventions a patient received over a longer period of time combined with a judgment of treatment effect. Arguably, there is no strong basis for our statistical assumptions but they seem rational to us. Therefore, we would like to keep the paper unchanged on this point. Please also see our reply to your query 6. We agree, therefore accuracy is assessed as a secondary outcome in this study. We realize that accuracy is only an indirect measure of improved quality of consultation and patient safety and will design further studies investigating this more directly, once this study showed a superiority of EDeR. We kept the manuscript unchanged. Thank you. We revised the Abstract accordingly. Well spotted thank you! Thank you we revised the paper accordingly. Thank you we revised the paper accordingly. Thank you we revised the paper accordingly. Thank you we revised the paper accordingly. Thank you we revised the paper accordingly.

14 25 Even if there is just one table provided in the paper, the authors should give it a number when referencing to it. 26 References related to the evaluation of medical informatics interventions are missing. 27 The selection of references according to the research question is arbitrary. 28 It is unclear whether a conflict of interest exists due to the involvement of the authors in the development of the EMR-system. Please include a statement related to the involvement of the authors. Thank you we revised the paper accordingly. Thank you we checked again and added the reference of your excellent 2008 Systematic Review published in J Biomed Inform. to the background section. We also added some additional references to underpin our statements better. Please see our reply to your query 26 Thank you we fully agree and amended the paper accordingly. We declare to have no conflict of interest. (p.13, last para) VERSION 2 REVIEW REVIEWER REVIEW RETURNED Prof. Dr. med. Jürgen Stausberg Ludwig-Maximilians-Universität München Institut für Medizinische Informationsverarbeitung, Biometrie und Epidemiologie (IBE) Germany 05-Mar-2013 GENERAL COMMENTS The authors submitted a revised manuscript of their study protocol that takes into account the majority of the reviewer s concerns. Only a few points remain. The authors may or may not change their study protocol according to those points or may or may not comment on those points in their manuscript. A) The reviewer expects a bias favoring EDeR due to the following reasons: 1) The authors compare real-world paper-based records with artificial electronic records. Therefore, the artificial electronic records will be more complete, plausible and unambiguous than the paperbased ones. However, competing study designs suffer also from weaknesses due to the impossibility to create a double-blind setting for that type of analysis. 2) Voluntarily participants for technological studies tend to prefer the technological alternative. 3) Training in EDeR will be a relevant confounder in this study, maybe in the opposite direction at the first time point. B) There is some literature about the methodology of evaluating medical informatics technologies, e. g. Nykänen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart- Zephir MC, Ammenwerth E. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform. 2011; 80: Brender J. Handbook of Evaluation Methods for Health Informatics. Elsevier, Friedman CP, Wyatt JC. Evaluation Methods in Biomedical

15 Informatics. Springer, Wyatt JC. Clinical data systems, Part 3: Development and evaluation. Lancet ; 344: C) Figure 1 is not helpful because it does not appropriately represent the flow of the study. For example, the box patient model from original paperforms describes something that should be available before the first time point, but it is palced in the middle of the 12 months. There should be an unambiguous flowchart presenting the timeline and the logic of the study. VERSION 2 AUTHOR RESPONSE # Query Our reply 1 The authors submitted a revised manuscript of their study protocol that takes into account the majority of the reviewer s concerns. Only a few points remain. The authors may or may not change their study protocol according to those points or may or may not comment on those points in their manuscript. 2 A) The reviewer expects a bias favoring EDeR due to the following reasons: 1) The authors compare real-world paper-based records with artificial electronic records. Therefore, the artificial electronic records will be more complete, plausible and unambiguous than the paper-based ones. However, competing study designs suffer also from weaknesses due to the impossibility to create a double-blind setting for that type of analysis. 2) Voluntarily participants for technological studies tend to prefer the technological alternative. 3) Training in EDeR will be a relevant confounder in this study, maybe in the opposite direction at the first time point. Thank you for your additional queries. We are grateful for the comments and will address those in a point-to-point manner and revise the paper accordingly. The reviewer addresses an important issue related to the planned study: We understand the concern but do not fully agree with point 1: We are not creating artificial electronic records but exact representations of an existing paper-based record. Thus, the content of each pair of paper and electronic record is identical. The only difference is one little contrast that is changed for a specific task. We realize that this was not made fully clear and amended the protocol as follows (p.8, 1. para): The electronic records are exact representations of an existing paperbased record. Patient records differ only in respect to one contrast that is changed for a specific task. Thus, besides that task-specific detail, the remaining content of each pair of patient records is identical. Concerning 2: we fully agree and added this to the discussion section (p.12, 2. para): It can be argued that choosing volunteers for this study could lead to exaggerated results. Nevertheless we believe that the results of our study will provide a rational basis to quantify the added value of an electronic medical

16 record system in an ophthalmologic environment. Concerning 3: We do not fully understand why confounding should occur. We are assessing differences in effects between the two time-points. This allows us examining differences in training on the relative efficacy of the electronic patient record. Therefore, we would like to keep the protocol unchanged. 3 B) There is some literature about the methodology of evaluating medical informatics technologies, e. g. Nykänen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC, Ammenwerth E. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform. 2011; 80: Brender J. Handbook of Evaluation Methods for Health Informatics. Elsevier, Friedman CP, Wyatt JC. Evaluation Methods in Biomedical Informatics. Springer, Wyatt JC. Clinical data systems, Part 3: Development and evaluation. Lancet ; 344: C) Figure 1 is not helpful because it does not appropriately represent the flow of the study. For example, the box patient model from original paperforms describes something that should be available before the first time point, but it is placed in the middle of the 12 months. There should be an unambiguous flowchart presenting the timeline and the logic of the study. We would like to thank the reviewer very much for pointing at additional pertinent literature. We will study these references carefully and will set them into perspective when writing-up the paper based on our findings. We agree and revised the figure accordingly.