Author's response to reviews Title:Online training course on critical appraisal for nurses: adaptation and assessment Authors: Eva Reviriego (ereviriego@ej-gv.es) María Ángeles Cidoncha (mariaangeles.cidonchamoreno@osakidetza.net) José Asua (jasua@ej-gv.es) Marie-Pierre Gagnon (Marie-Pierre.Gagnon@fsi.ulaval.ca) Maider Mateos (evaluacionosteba@ej-gv.es) Lucía Gárate (lucia.garateechenique@osakidetza.net) Elena De Lorenzo (elena_delorenzo@ehu.es) Rosa María González (rosamari.gonzalezllinares@osakidetza.net) Version:2Date:17 June 2014 Author's response to reviews: see over
BMC Medical Education MS: 9484364671030908 Title of the paper: Online training course on critical appraisal for nurses: adaptation and assessment Authors: Eva Reviriego, María Ángeles Cidoncha, José Asua, Marie-Pierre Gagnon, Maider Mateos, Lucía Gárate, Elena De Lorenzo and Rosa María González Submission date: 17 th of June, 2014 Dear Editor and Reviewers: Thank you for considering the manuscript entitled "Online training course on critical appraisal for nurses: adaptation and assessment" for publication in BMC Medical Education. We have reviewed all suggestions and recommendations of the two reviewers, and made many changes in the new version of the manuscript. Guided by your helpful comments we believe that we have considerably improved the manuscript. Please find below our point-by-point responses to each of the reviewers comments. We have grouped them according to the different sections of the manuscript, and each reviewer. We have also highlighted in yellow the modifications made from the original manuscript. Yours sincerely, Eva Reviriego Researcher Osteba, Basque Office for Health Technology Assessment Ministry for Health, Basque Government C/ Alameda Rekalde 39, 6º 48008 Bilbao Spain Tel. +34 944032429 Email: ereviriego@ej-gv.es
Author s responses to reviewers comments TITLE - Reviewer 2: The title refers to adaptation but the study presents limited rigorous methodology and no result on this process. An exhaustive description of the translation process has already been published in reference 31; this is cited in the new version of the manuscript. ABSTRACT - Reviewer 2: Abstract adequate in length and well structured. Content was adequate with the exception of the conclusions which are too general and need to relate more specifically to the study. We have added the following paragraph: Participants significantly improved their knowledge score and self-directed learning readiness after the educational intervention and they were overall satisfied with the course. LANGUAGE: - Reviewer 2: With due respect to the authors where it is apparent that English is not their first language there are concerns about the writing style; this needs to be tightened from both grammatical and scientific writing perspectives. The manuscript was translated through professional services offered by a team knowledgeable in health sciences including native English speakers. We passed your comment to them; they have revised the English and, with their help, we have extensively reworked the writing in several parts of the manuscript (especially the Results section). We believe that the use of English and quality of scientific writing in our manuscript has now improved. BACKGROUND - Reviewer 1: 1.First, as I read the article there was some confusion regarding the purpose. See info on general objective and the hypotheses. It seems there are two purposes: 1) Increase knowledge of nurses in the use of critical appraisal
skills for use of research and 2) assess readiness for the use of e-learning as a learning platform and 3) to establish the satisfaction with the course, both content and methods. I think this needs to be clear. This is really not a hypothesis testing studying. It is really more exploratory and descriptive. So you want to have clear objectives and purpose and could have aims but don t need hypotheses. We have modified the text about our general and specific objectives. And we have deleted the paragraph about the hypothesis, agreeing that this approach is not pertinent to this study. - Reviewer 2: Pg. 3There are concepts that need to be tightened eg in paragraph 1, where evidence based practice should include not just personal experience but more specifically clinical experience or clinical expertise. In paragraph 2, differentiate a review of the literature form a systematic review. In paragraph 3, there needs to be a more critical and focussed evaluation on e-learning. Use the cited references 21-23 and other if needed to present a more scientific perspective. The paragraphs mentioned have been modified in response to the reviewers comments. METHODS - Reviewer 1: Second methods issue is about the translation of instruments. Usually you want the instrument translated to the new language (Spanish) by bi lingual persons and then you want to a different person who is bilingual (Spanish and French) to back translate it to do a more thorough check on capture of the key concepts. I would have also like you to have reported the psychometrics of the original instruments so we could compare if anything was lost or perhaps gained. - Reviewer 2: Pg. 5:There needs to be more detail provided on the process of translation of the original Laval InfoCritique program into Spanish and then to Castilian Spanish, as there strong potential for errors and bias in this process.
Were the translated programs pilot-tested and evaluated? What were the results? What concepts required further elaboration? The translation and adaptation process of the Infocritique modules has been described in details elsewhere (ref. 31). - Reviewer 2: Pg. 5: What was the eligibility criteria for inclusion? Pg. 5: target population with basic knowledge of research methodology. Is not sufficient information. There needs to be a tighter definition of previous exposure that will be acceptable for inclusion. We have added the eligibility criteria that were used: the nurses had to be working and have completed an academic degree in Nursing. - Reviewer 2: Pg. 6: Present detail on the role of the three tutors in the intervention. We have added the following paragraph: Tutors were responsible for approving the course content, answering questions and monitoring progress of students through the course, and compiling a report on students questions after completion of the course. - Reviewer 2: Pg. 5-6: Further information is needed on the InfoCritique program developed by Laval University Faculty of Nursing. While a brief overview was given of the time commitment of students and the supporting resources, there needs to be more detail presented on the structure and content of the online course. We have added the following information: The InfoCritique program trains students in the critical appraisal of scientific literature and assesses them based on tests focusing on the content of each module. The four selfdirected learning modules available in Spanish cover 1- clinical trials; 2- systematic reviews; 3-diagnostic tests; and 4- qualitative studies.
Every module begins with some basic information about the type of research study under consideration and associated methods. Then, there are one or more exercises focused around a clinical case. For each clinical case, students are presented with a scientific article reporting a study of the design covered in the module. Students have to read the article and use a critical appraisal grid adapted to each study design in order to appraise the study. Each module also provides interactive links to supplementary information, such as a glossary and additional references. Finally, students have to complete an assessment test and a satisfaction form for each module, which is used as feedback for the tutors [32]. - Reviewer 2: Pg7: For completeness indicate in the body of the text that there were also questions relating to socio-demographic data which were asked. (these questions were included in the appendix.) We have added the following information: A specific questionnaire was also designed to collect data on participants sociodemographic characteristics, covering gender, age, year of completion of their first nursing diploma, level of education, number of hours devoted to research training, whether they put into practice methodological research knowledge, and current position. (Additional file 4). - Reviewer 2: Pg. 6-7: Further detail and clarification of the outcome measures is required. How were the 20 questions chosen from the four modules for the knowledge assessment questionnaire? Were a specific number of questions drawn from each module? These 20 questions were randomly selected from the final assessment tests included in each of the 4 modules, 5 questions being taken for each module. - Reviewer 2: Pg 7: Indicate the number of items in each of the subscales of the SDLRSNE. We have added the following information:
The questionnaire, composed of 40 items divided into 3 subscales, Self-management (13 items), Desire for learning (12 items) and Self-control (15 items), was administered before and after the course. - Reviewer 2: What did the authors consider prospectively would constitute a potentially meaningful point score change in the SDLRSNE.Are there previous studies in nursing that have considered this? (For example a study in pharmacy students using the self-directed learning for readiness scale the authors considered a 15-point change in the SDLRS score to be potentially meaningful based on previously collected data in pharmacy students: Huynh D, Haines ST, Plaza CM, Sturpe DA, Williams G, Rodriguez de Bittner MA, Roffman DS. 2009, The impact of advanced pharmacy practice experiences on students' readiness for self-directed learning. Am J Pharm Educ. 2009 Jul 10;73(4):65.) We did not prospectively consider a meaningful point score change in the SDLRSNE and we are not aware of such proposal in the nursing literature. However, we acknowledge that it would be a good idea for future research (see Limitations). - Reviewer 2: Who undertook the translation of the Self-Directed Learning Readiness Scale for Nursing Education (SDLRSNE) and how was rigour established in this process? How was the newly developed translated questionnaire tested prior to the study? We have added the following information: In this research, the self-directed learning readiness was measured using the 40 items of the SDLRSNE rated on 5-point Likert scales ranging from strongly disagree to strongly agree. The Spanish version of the SDLRSNE scale was validated following the steps proposed by Vallerand for transcultural validation [36].
- Reviewer 2: Pg 8: Qualify that reliability here refers to internal consistency as compared to other forms of reliability eg test-retest reliability (here and in results on pg 11) Yes, it does. We changed the term reliability to internal consistency, that is, it is only a measure of the reliability of an instrument. - Reviewer 1: Ethical considerations: Was there review by a Human Subjects Board to assure protection of human subjects? You mentioned some of the steps taken to protect privacy and to withdraw but I did not see that it had the approval of the agency review board. Again, maybe it is not an issue in Spain. - Reviewer 2: I have some concern about the statement on page 9 Ethical approval was not required given that the study was an evaluation of a learning activity. While this may vary from country to country the Australian National Health and Medical Research Council (NHMRC) Australian Code for the Responsible Conduct of Research. While educational data can be collected without ethical approval as part of a course audit, ethical approval is required if the study is to be published. Can the editorial committee of the BMC Medical Education please comment in this aspect of the manuscript. As the reviewer mentions, the ethical approval is not required for a learning activity in our jurisdiction, but as there is no clear guidance on the ethical requirement for publishing, we have sought ethical approval. We have attached the decision of the Clinical Research Ethics Committee.
RESULTS - Reviewer 2: Remove results from each of the sub-headings for the areas of results. We have modified the sub-headings accordingly. - Reviewer 1: I could not see where you reported the level of education of the nurses. Perhaps that is because there are not so many ways to be an RN in Spain. Were these all nurses with undergraduate degrees in nursing or did some have graduate education and how much nursing experience did they have. Did you look at the readiness for e-learning by age? Was that a factor? We added the following information: All the nurses had an academic degree in Nursing and 84% of the sample worked in a clinical setting, 12% in management and 4% in teaching.
We did not analyze the correlation between age and readiness for e-learning, but it could be very interesting issue to explore in future studies. - Reviewer 2: These results could be presented more succinctly. Avoid repeating what Figure 1 shows by combining the pre-and post- results more. Avoid repetition of the methodology eg that the McNemar test will be used to analyse the difference in the pre-and post-scores. The knowledge results could be presented under one heading rather than separation into a heading for pre-course results and another heading for post-course results. It would be useful either here or in the discussion to have some indication of what was included in the items which scored both well and poorly and those in which the greatest change was made, rather than just include the numbers of these items. We have added more specific details about the items and the corresponding text now reads as follows: As shown in Figure 1, there were some issues about which participants already had considerable knowledge prior to the study. In particular, more than 90% of them gave correct answers to questions such as In a systematic review, researchers should always try to identify differences between the studies considered (item 6), the sample size is important in the evaluation of the validity of a test (item 15), and items related to qualitative research (items 16, 17 and 18). In other areas, however, less than half of the participants gave correct answers. For instance, participants had gaps pre-course gaps in their knowledge for questions 2, 8, 11 and 12 related to epidemiological concepts (significance or sensitivity and specificity). We have deleted the sentences about McNemar and other details of the statistical tests in the Results. We have extensively reworked the text to present all results more succinctly. - Reviewer 2: Cronbach s alpha was used to test the internal consistency of the complete questionnaire and for the three subscales; the subscales however were not used in the results. Can you clarify why the internal consistency of the subscales were tested? Alternatively it may be useful to present the results for the subscales.
When Cronbach's alpha is used on a scale that is composed of several dimensions, as in this case, it is possible to underestimate the internal consistency. Therefore, it is desirable to also present the values of Cronbach's alpha for the different dimensions. Our interest was to analyze the data from the completed questionnaire. The idea was not to analyze the data of the subscales that we only used to assess the validity of the questionnaire. [33]. Fisher MJ, King J: The Self-Directed Learning Readiness Scale for nursing education revisited: a confirmatory factor analysis. Nurse Educ Today. 2010 Jan;30(1):44-8. DISCUSSION - Reviewer 2: The discussion is short, (less than 400 words) and lacks a clear structure. The discussion needs to present a more critical and thoughtful interpretation of the findings. There is no discussion of the findings in relation to other comparable studies ie present new findings in the context of previous findings. P 14.paragraph 4: While general comments were made about future directions, these need to be more specific The Discussion section has been rewritten. LIMITATIONS - Reviewer 1: I think there needs to be a limitations section. While you had excellent pre/post results this is the weakest design and could have been strengthened by adding a second post- test which would help you know better about retention of material. It also may have been helpful at second pos0t test time to ask for examples of how they had used the newly gained knowledge. We have included a limitations section. - Reviewer 2: Discuss here or in discussion the 5point increase in SDLRSNE and whether the change which was statistically significant, was also considered meaningful
In line with this comment, we have added this sentence in the Discussion: Finally, we did not set a threshold for a meaningful point score change in the SDLRSNE score a priori. In future studies, it would be interesting to establish such criteria to assess whether the differences between pre- and post-intervention scores are important in practice. CONCLUSIONS - Reviewer 1: Conclusion: Lastly, I would like you to strengthen the conclusions page. You have some good findings that do address your purposes. Need to discuss these more thoroughly in a paragraph not just bullets and then identify how you would modify the course and delivery based on the results. This would also be a good place to discuss the limitations. - Reviewer 2: These need to relate more specifically to the aims of the study which were 1) to assess the applicability of an online training course on critical appraisal developed by the Faculty of Nursing at Laval University, Quebec, and adapt it to our healthcare context; and 2) to explore the impact of this online training course in terms of knowledge acquired and self-learning readiness as well as satisfaction with the program. What was presented related to the second objective; summarise how much knowledge and self directed learning ability changed. We have modified the Conclusion section. REFERENCES Reviewer 2: Pp18-20 Minor errors in reference list when assessed against Instructions for Authors; use colon after initial of last author; issue number to be bolded. In text references should use a square not a round bracket. The References section has been modified.
TABLES, FIGURES AND ADDTIONAL FILES - Reviewer 2: Format and presentation Ensure that all table sand figures are referred to within the body of the text in addition to indicating the position of the table/ figure within the text. All five figures will need to be printed in colour as there is loss of clarity of the ratings in black and white. The questionnaire in the appendix (Knowledge Questionnaire, Self-Directed Learning Readiness Scale for Nursing Education (SDLRSNE) and Satisfaction Questionnaire) would be better as three or four separate appendices. We have accepted your suggestions related to format and presentation; and we have divided the questionnaires into four. OTHER: - Reviewer 2: Avoid use of the term project where you mean study. Revised and changed.