PEER REVIEW HISTORY ARTICLE DETAILS TITLE (PROVISIONAL)

Transcription

1 PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form ( and are provided with free text boxes to elaborate on their assessment. These free text comments are reproduced below. ARTICLE DETAILS TITLE (PROVISIONAL) AUTHORS PubMed search filters for the study of putative outdoor air pollution determinants of disease Curti, Stefania; Gori, Davide; Di Gregori, Valentina; Farioli, Andrea; Baldasseroni, Alberto; Fantini, Maria Pia; Christiani, David; Violante, Francesco S; Mattioli, S VERSION 1 - REVIEW REVIEWER REVIEW RETURNED Andreas Seidler Institute and Policlinic of Occupational and Social Medicine, Faculty of Medicine, TU Dresden, Germany 13-Jul-2016 GENERAL COMMENTS This paper aims to identify efficient PubMed search filters for the study of diseases related to environmental outdoor air pollution. To assess efficiency, the sensitivity, specificity and the number needed to read abstracts is calculated with the studies included in methodologically adequate systematic reviews as reference. The authors develop a more sensitive and a more specific search string. The methodological approach is convincing, and the developed ready-to-use -tool can easily be applied by persons trying to get an overview over (new) papers on the health effects of outdoor air pollution. The paper is clearly written and well organized. However, it should be made somewhat clearer, who will benefit from this publication resp. the developed tool: - Persons (scientists, decision-makers, etc.) who are from time to time interested in new studies on the health effects of outdoor air pollution could easily apply this ready-to-use tool. Applying the more specific search string would allow for a very quick ( efficient ) search, but would only identify about 80% of the relevant articles. So maybe it would allow for a better overview over new evidence on the health effects of air pollution to just monitor newly published systematic reviews than to periodically read a (random?) selection of about 80% of new primary studies? - Scientists planning to conduct a systematic review on the health effects of outdoor air pollution could use the more sensitive search string. However, a systematic review should not be based on only one database. Therefore, the sensitive search string would have to be translated to other databases (e.g. Embase) which seems to be rather time-consuming because of the length and complexity of the sensitive search string. Moreover, the characteristics (particularly sensitivity) of this translated search string are unknown. - Scientists planning to conduct a systematic review on another topic could try to develop their own search string according to the procedure proposed by Curti et al. However, as this procedure seems rather complex and time-consuming, the question remains: will the gain in completeness justify the considerable effort? To

2 answer this question, it would be interesting to compare the characteristics (sensitivity, specificity, number needed to read abstracts) of the sensitive search string with the characteristics of the search string applied in conventional systematic reviews. The authors should discuss these aspects in more detail. Minor remarks: - The sensitivity of the specific search string and the specificity of the sensitive search string should be given in the abstract. - The Methods section could be shortened. REVIEWER REVIEW RETURNED Sarah Hayman Flinders University, Adelaide, Australia 01-Aug-2016 GENERAL COMMENTS I found this a very interesting and worthwhile project and paper, with an outcome of a useful resource. I have made some small suggestions about the paper and asked some minor questions for clarification relating to the methods. They are in the attached document as comments for consideration by the authors and editors. The reviewer also provided a marked copy with additional comments. Please contact the publisher for full details. VERSION 1 AUTHOR RESPONSE Reviewer: 1 Reviewer Name: Andreas Seidler Institution and Country: Institute and Policlinic of Occupational and Social Medicine, Faculty of Medicine, TU Dresden, Germany Please state any competing interests: None declared Please leave your comments for the authors below This paper aims to identify efficient PubMed search filters for the study of diseases related to environmental outdoor air pollution. To assess efficiency, the sensitivity, specificity and the number needed to read abstracts is calculated with the studies included in methodologically adequate systematic reviews as reference. The authors develop a more sensitive and a more specific search string. The methodological approach is convincing, and the developed ready-to-use -tool can easily be applied by persons trying to get an overview over (new) papers on the health effects of outdoor air pollution. The paper is clearly written and well organized. However, it should be made somewhat clearer, who will benefit from this publication resp. the developed tool: - Persons (scientists, decision-makers, etc.) who are from time to time interested in new studies on the health effects of outdoor air pollution could easily apply this ready-to-use tool. Applying the more specific search string would allow for a very quick ( efficient ) search, but would only identify about 80% of the relevant articles. So maybe it would allow for a better overview over new evidence on the health effects of air pollution to just monitor newly published systematic reviews than to periodically read a (random?) selection of about 80% of new primary studies? We really thank the reviewer for the comment. We have now added the following sentences in the discussion section: On the one hand, 54% of the abstracts retrieved by the more specific filter provided information on

3 environmental determinants of disease related to outdoor air pollution (NNR 1.9), suggesting that this search filter can be applied to etiological questions encountered in routine practice. In particular, the more specific filter would allow researchers and professionals for a better overview over new evidence on the health effects of outdoor air pollution by monitoring newly published systematic reviews. - Scientists planning to conduct a systematic review on the health effects of outdoor air pollution could use the more sensitive search string. However, a systematic review should not be based on only one database. Therefore, the sensitive search string would have to be translated to other databases (e.g. Embase) which seems to be rather time-consuming because of the length and complexity of the sensitive search string. Moreover, the characteristics (particularly sensitivity) of this translated search string are unknown. We thank the reviewer for pointing out this. We have now modified the limitations section including the following sentences: Even though it has been reported that most of the high quality articles, like those included in Cochrane Reviews, are indexed in PubMed[27], it is strongly recommended to consult more than one relevant database when performing systematic reviews of the literature.[28] The present study was restricted to PubMed as any medical database has its own rules (e.g. syntax, map of key terms) and needs to be studied separately (including the assessment of sensitivity and specificity). When carrying out a systematic review on the health effects of outdoor air pollution, it could be possible to translate the proposed more sensitive filter into other database syntax; however, it has to be underlined that it is likely to suppose that the characteristics of the translated filter - in terms of sensitivity and specificity - would not be the same as those calculated for PubMed. - Scientists planning to conduct a systematic review on another topic could try to develop their own search string according to the procedure proposed by Curti et al. However, as this procedure seems rather complex and time-consuming, the question remains: will the gain in completeness justify the considerable effort? To answer this question, it would be interesting to compare the characteristics (sensitivity, specificity, number needed to read abstracts) of the sensitive search string with the characteristics of the search string applied in conventional systematic reviews. The authors should discuss these aspects in more detail. We really thank the reviewer for these helpful comments. We have now compared the characteristics (sensitivity, specificity and NNR) of the proposed search filters with those of a conventional search filter developed in a recent systematic review on outdoor air pollution. Although our procedure seems to be rather complex and time-consuming, the proposed search filters provide a very efficient tool to retrieve more pertinent articles and appear to be highly sensitive - two essential characteristics when conducting a systematic review. We have now addressed this topic and added a new section called Comparative analysis. We have now modified the methods section as follows: Comparative analysis We evaluated whether the method we used to formulate the proposed search filters could be efficiently carried out by other researchers when creating new filters on another topic. For this purpose, we compared the characteristics (sensitivity, specificity and NNR) of the proposed search filters with those of a conventional search filter developed in a recent systematic review on outdoor air pollution. We searched Medline using a mini-filter composed by the core terms and adding the terms (systematic rev* OR metanal* OR meta anal* OR metaanal*) limiting the search to abstract availability. Two authors (SC; SM) independently screened titles and abstracts of the most recent references to select that systematic review that addressed a specific medical condition

4 potentially related to outdoor air pollution and published on a top-ranked journal (classified as Q1 or Q2). The references retrieved by the conventional search filter of the recent systematic review (using only the part of the filter on outdoor air pollution) were compared to the gold standard identified with the same methodology described in the previous section. Sensitivity and specificity along with 95% CI produced with the Wilson score method[20] were then calculated. Furthermore, we assessed the performance of this conventional filter on outdoor air pollution in the study of the same three diseases mentioned above (i.e. arrhythmia, sudden death and congenital heart defects). The same methodology for the evaluation of the pertinence of the retrieved abstracts was applied and NNR values were then calculated. All the searches were run on 22nd September We have introduced the following section in the results as well: Comparative analysis We screened the titles and abstracts of the most recent references of potentially eligible systematic reviews on outdoor air pollution. We were able to identify the most recent one that satisfied the inclusion criteria. The selected systematic review evaluated the association between cognitive functioning and exposure to air pollution and it was published on a journal ranked Q1 in the public, environmental and occupational health category [Clifford 2016]. The already identified gold standard was used as a comparison set for the conventional search filter on outdoor air pollution developed in the selected systematic review. This filter correctly identified 175 pertinent references out of 206 and 1839 out of 2741 not pertinent ones. The sensitivity of this conventional filter on outdoor air pollution was 85.0% (95% CI, ) together with a specificity of 67.1% (95% CI, ). With respect to the assessment of the NNR values, this conventional filter on outdoor air pollution developed in a recent systematic review retrieved 79 articles for arrhythmia, 42 for sudden death, and 22 for congenital heart defects. Out of the 143 articles retrieved in the overall search, 57 were judged as pertinent, with an NNR of 2.5. We have added the following section in the discussion: In comparison with the proposed more sensitive filter, the conventional one on outdoor air pollution developed in a recent systematic review lacks in sensitivity (98.5% for the more sensitive filter vs 85.0% for the conventional one) and reports characteristics more similar to the more specific one. The conventional filter on outdoor air pollution developed in the selected systematic review retrieved only 143 articles for the investigation of three diseases selected a priori. Of these, 57 were judged as pertinent (NNR 2.5). Specifically, the number of retrieved abstracts and of pertinent ones were found to be substantially lower than those calculated for the more sensitive filter (895 retrieved, 271 pertinent abstracts) and even lower than those of the more specific one (260 retrieved, 140 pertinent abstracts) (Table 2). It s striking to note that the conventional filter of the selected systematic review reported roughly the same sensitivity of the proposed more specific one, probably because of the conventional way to develop search filters used by the authors of the six systematic reviews included in the gold standard. On the other hand, the conventional filter developed in the selected systematic review explores only a limited subset of the total amount that should be explored, as shown when assessing the NNR of the three diseases selected a priori. Although our procedure seems to be rather complex and time-consuming, the proposed search filters provide a very efficient tool to retrieve more pertinent articles and appear to be highly sensitive - two essential characteristics when conducting a systematic review. Minor remarks: - The sensitivity of the specific search string and the specificity of the sensitive search string should

5 be given in the abstract. This has now been reported in the text of the abstract. - The Methods section could be shortened. We know that the methods section is pretty long, but considering the comments of the reviewers, we were forced to add more explanations and analysis in the manuscript. Moreover, since the procedure is rather complex, shortening the methods section would create further difficulties for the readers. Reviewer: 2 Reviewer Name: Sarah Hayman Institution and Country: Flinders University, Adelaide, Australia Please state any competing interests: None declared Please leave your comments for the authors below I found this a very interesting and worthwhile project and paper, with an outcome of a useful resource. I have made some small suggestions about the paper and asked some minor questions for clarification relating to the methods. They are in the attached document as comments for consideration by the authors and editors. ** The application was unable to attach manuscript files to this , because one or more of the files exceeded the allowable attachment size (10MB). ** Comment xx1 Might be clearer to say something like increasingly popular in the press including medical literature as evidenced by indexing in medical databases. We thank the reviewer for the comment. We have now changed the sentence reported in the introduction section as follows: Topics related to environmental exposure have become a major issue for public health throughout the world and increasingly popular in the press including medical literature as evidenced by indexing in medical databases. Comment xx2 Are We have now modified the first sentence of the methods section as follows: The four stages of search filter development are summarized in Figure 1. Comment xx3 Was this randomly collected? We thank the reviewer for pointing out this. We have now specified that the sample was systematically recruited, that is not, strictly speaking, a random sample. We have modified the sentence of the methods section as follows: When the search query produced more than 1,000 citations, we collected a systematically recruited sample of 100 abstracts. Comment xx4 across the might be clearer This has now been modified accordingly. Comment xx5

6 Should be of every. This has now been modified accordingly. Comment xx6 I query in general the inclusion of all candidate terms in the sensitive filter. I see that the terms in the specific filter were only included if they passed a performance threshold which introduced a measure of objectivity into the selection of the terms. However in the sensitive filter, all potential terms appear to have been included, regardless of their recall performance. Theoretically any term could therefore be chosen and included in the sensitive filter, making the testing process redundant for this filter. I would find it useful if there could be included an explanation of why all candidate terms were included in the sensitive filter. The reviewer is right. In the more sensitive filter we included all the tested terms we examined. However, the selection of these terms follows a rather complex process (described in the first part of the methods section) that does not mean that every proposed term would have been examined. In particular, the testing process was necessary to identify those terms (threshold of 40% of pertinent abstracts) that could be included in the more specific filter. We included all the tested terms in the more sensitive one, because every term contributes to the retrieval of the pertinent literature, even if with different proportion of pertinent abstracts. We have added the following sentences in the methods section: We then included only the terms with a proportion of pertinent abstracts 40% in the more specific search filter. All the tested terms were instead included in the more sensitive one, since every term contributed to the retrieval of the pertinent literature, even if with different proportion of pertinent abstracts. Comment xx7 Suggest omit this word or replace with Then or Thence. The word Therefore does not seem correct here. Thank you for the suggestion. The word Therefore has now been omitted from the sentence. Comment xx8 Was this truncated to congenital heart defect*? We completely agree with the suggestion proposed by the reviewer. We have now changed the search of this disease as (congenital heart defects [MH] OR congenital heart defect*) and modified the results accordingly (see Table 2). Please note that only the total number of abstracts indexed for congenital heart defects was slightly different. Comment xx9 In general, throughout the project, were the authors who undertook evaluation of pertinence also involved in devising the searches and selecting search terms? This might be a potential source of bias. For the present study, we decided to divide the different tasks between pairs of authors and another author was in charge to resolve any disagreements. And this is a standard practice when carrying out systematic reviews (even Cochrane reviews). Moreover, the authors have not a propensity for one term with respect to another one. So, we would argue that this should not be a major source of bias. Comment xx10 Were these non-mesh terms as well as MeSH? Useful to make clear here.

7 We thank the reviewer for this suggestion. We have now specified as follows: This list was extended by non-mesh terms retrieved in pertinent articles, keywords found in reports on air pollution of the EPA and WHO, and terms suggested by co-authors. Comment xx11 Was text-mining used as an additional more objective way to identify non-mesh (and indeed also MeSH) terms? Textual analysis can be done of titles, abstracts and keywords in a gold standard set of references We thank the reviewer for the relevant comment. We have not used any computational technique (like text-mining), since our selection of search terms was to some extent arbitrary, but at the same time so extensively performed. We have modified the discussion section as follows: Our selection of non-mesh terms was to some extent arbitrary, but at the same time it was so extensively performed that the identification of such search terms with computational techniques (like text-mining) was not considered worthwhile. In addition, considering the specificity and sensitivity values of the proposed filters together with NNR values, this a priori limitation did not appear to greatly affect the end product. Comment xx12 Could note here (as you have done elsewhere) that a url for each search can be provided for even greater clinical utility. We thank the reviewer for the suggestion. We have now added the following sentence in the conclusion section: Both filters can be copied and pasted into the PubMed search box together with the name of the disease under study. Alternatively, the filters can be evoked in PubMed by entering the shortened URLs (Uniform Resource Locators) provided at the bottom of Table 1.