PEER REVIEW HISTORY ARTICLE DETAILS TITLE (PROVISIONAL)

Transcription

1 PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form ( and are provided with free text boxes to elaborate on their assessment. These free text comments are reproduced below. ARTICLE DETAILS TITLE (PROVISIONAL) AUTHORS safetxt: a pilot randomised controlled trial of an intervention delivered by mobile phone to increase safer sex behaviours in young people McCarthy, Ona; French, Rebecca; Baraitser, Paula; Roberts, Ian; Rathod, Sujit; Devries, Karen; Bailey, Julia; Edwards, Phil; Wellings, Kaye; Michie, Susan; Free, Caroline VERSION 1 - REVIEW REVIEWER REVIEW RETURNED Brian Suffoletto University of Pittsburgh USA My competing interests are that I design and test text message behavioral interventions. I also receive royalties form a company who licensed one of our text message interventions. 02-Jul-2016 GENERAL COMMENTS This manuscript describes findings from a pilot trial of 200 young adults recruited either on-site at sexual health clinics or via phone for patients who had been seen at a sexual health clinic in England and randomized to either a simple automated text message intervention or control messages, focusing outcome analysis primarily on acceptability of the intervention and feasibility of recruitment and follow-up. Authors report that they met their recruitment and followup goals and that the texting intervention was deemed acceptable to participants (which was collected at 1 month follow-up). Participants reported specifically that the messages made them think (reflect), were respectful, easy to understand, although a minority (30%) admitted that the messages made them take action in their lives. Authors report that from these findings they have started a main trial to examine cumulative incidence of STI at 12 months as their primary outcome and recruiting only those who have an STI at baseline. My main concerns: 1. Recruitment appeared to over-sample females, white participants and those without STD infections. Authors do not address how they will improve recruitment of men, those specifically with STI and other minorities. 2. There is a lack of description of the actual intervention. It is not clear how often it broadcasts messages, for how many days, whether there is any personalization or tailoring based on baseline or time-varying data or what participants get when they text back one of the keywords. Supplementary file 1 shows number of messages across time but there is no description of how frequency drops off (using some algorithm?). 3. Secondary (behavioral) outcomes are barely mentioned in

2 Results. Specifically, authors should mention that the effects of the texting intervention seem to be small, if at all, and related primarily to reductions in number of partners, but not condom use. Among those with an STI, the texting intervention may have effects on partner treatment. 4. If authors are going to use cumulative STI at 12-months as their primary outcome, they should state based on their pilot work how many participants would be needed to have at least 80% power to show a difference. 5. It would seem to me that given current ability to tailor and personalize computerized interventions, that the authors should have at least discussed the consideration of alternative texting designs or communication technology platforms that could improve effect sizes. The same could be said of content of messages, i.e. what messages seemed to resonate with participants. REVIEWER REVIEW RETURNED Dallas Swendeman University of California, Los Angeles Center for HIV Identification, Prevention, and Treatment Services. 11-Jul-2016 GENERAL COMMENTS Rev for Safetxt This is a well-written paper on an important topic. The methods for intervention development appear to be robust. The paper is nicely succinct. A few more details would be helpful from this reviewer s perspective, as outlined below. 1. In the existing research section, it would be helpful to know the general functions or content of the messages from the other studies, for example, did they incorporate reminders, educational information, etc? This information is only included for the third of the three studies cited. 2. Also in this section, citation 13 noting for the statement that results of mobile phone support for safer sex behaviors is equivocal appears to be a review for all health behaviors, not just sexual behaviors. Then the three studies that follow suggest the results of prior research are not equivocal. Clarify briefly, or include some studies with null results. 3. It is unclear where participants were recruited, until later in the methods. Consider moving the recruitment section earlier in the methods, perhaps before treatment and control group description or before outcomes section, or before eligibility criteria (and maybe moving outcomes). 4. In the safetxt intervention section, it would be helpful to know the 12 behavior change techniques included in the intervention, or at least the higher order techniques/functions from the taxonomy. If the authors are concerned about word limits, there is some redundancy between the second and fourth sentence that could be consolidated. In addition, it is not stated how many and how frequently messages were sent (this information is only provided for the control group). 5. Why was acceptability of intervention only assessed at 1-month follow-up instead of also at 3 and 12 months? One of the broader concerns in this field of mobile tech support is how sustainable

3 participation can be, which can be reflected in opinions on acceptability of messages over time (in addition to the studies stated primary outcome of follow-up retention). 6. For recruitment procedures it could be helpful to briefly note that the proportion of clients who enrolled out of those contacted is presented in the results? 7. The Masking section mentions in order to monitor the incoming text-messages, which implies that the intervention was bidirectional? Please clarify and describe the bi-directional functions if applicable. 8. The Masking section seems overly detailed to this reviewer, if word limits are a concern. 9. The Data collection and follow-up procedures section needs some attention to clarity. Why was follow-up response not assessed at 3 months? Both face-to-face and telephone recruitment were used, but it is unclear why and how? All participants enrolled by telephone had a positive test result. Some rationale or further explanation is needed. 10. Table 1, I think the variable domain titles should be placed at the top of each section rather than at the bottom. 11. The variables on last time participant had sex with some one new the participant/partner was tested prior to sex need some clarification details, perhaps in the text or maybe a footnote. What was the timeframe on these questions? Was there any clarifying language included? These could be confusing to respond to from this reviewer s perspective. 12. The primary outcomes section needs a bit of explanation in the text. I think maybe inserting assessment of between for and cumulative incidence of chlamydia would suffice. 13. Table 3 could be reformatted to make comparisons between 1 month and 12 month data easier, specifically, by making four columns for the data (control and intervention at 1 month and 12 month). 14. Also, I don t think the data on The participant read the messages we sent to the other person and how many the participant read is needed in the table since the data is sparse, the latter only applied to control group, and it is sufficiently described in the text. 15. It is not clear to this reviewer that the study was not powered for behavioral outcomes such as condom use or even treatment completion (I agree that the STI incidence would require much larger sample in the context of treatment settings). The sample size is modestly large for a pilot. The pilot protocol states that one of the aims is to assess behavioral outcomes. The supplementary table (file 4) is not set up to make comparisons over time only comparisons between intervention and control are presented for each time point. Comparisons should be presented for Month 1 and 12 by intervention and control (and baseline ideally, was there a baseline?). It seems that there could be some statistically significant differences by intervention group over time for condom use at last

4 sex (control had declined rate over time, intervention had increased rate over time, but there is the attrition issue to contend with). It is fine that this paper focuses on acceptability and process outcomes, but the behavioral outcome data in the supplementary files is distracting and not setup correctly to assess whether or not there were behavioral impacts, and in what direction those could have been. I think it would be better to either exclude the sexual behavior results completely (even from supplementary files), or do the analysis appropriately and state clearly that there were no significant intervention effects, then focus the limitation section on why that might be the case (and also include the supplementary file information). All the content on masking and blinding in the limitations is much less relevant than the implications of sample size, statistical power, and assessment methods for behavioral outcomes. 16. Comparison with other studies section is okay, but the interpretation is compromised by not including information on incentives (a key factor in participation and retention rates). Large scale surveys very often have lower response rates than smaller scale behavioral intervention trials. In addition, it is not clear how the rates presented for the other studies related to the initial enrollment rate of this study. Also, the first and last sentences of this section seem to contradict each other (i.e., higher follow-up vs. compares well with similar studies ). 17. What additional follow-up strategies will be developed or are being considered or can be suggested? Since this is a primary topic of this paper, it would seem more relevant to discuss this topic than to simply end on more will be developed. VERSION 1 AUTHOR RESPONSE Reviewers' Comments to Author Reviewer: 1 1. Recruitment appeared to over-sample females, white participants and those without STD infections. Authors do not address how they will improve recruitment of men, those specifically with STI and other minorities. We have now addressed this on pg. 10 Implications for the main trial: As the target group in the main trial differs from the pilot trial, we will monitor recruitment to enable us to develop strategies to recruit any groups that are under-represented. In order to achieve a more demographically diverse sample, we are also recruiting from clinics with greater numbers of male clients that reporting having sex with men. 2. There is a lack of description of the actual intervention. It is not clear how often it broadcasts messages, for how many days, whether there is any personalization or tailoring based on baseline or time-varying data or what participants get when they text back one of the keywords. Supplementary file 1 shows number of messages across time but there is no description of how frequency drops off (using some algorithm?). We have added additional clarification of the intervention (pg. 5 the safetxt intervention). This section now includes a description of the intervention functions, the time frame of message delivery and a breakdown of the tailored messages sets.

5 3. Secondary (behavioral) outcomes are barely mentioned in Results. Specifically, authors should mention that the effects of the texting intervention seem to be small, if at all, and related primarily to reductions in number of partners, but not condom use. Among those with an STI, the texting intervention may have effects on partner treatment. We have removed the report of the behavioural outcomes from this paper. We had discussed this previously, and in further discussion considering both reviewer s comments, we agreed that it s best not to include the behavioural outcomes. 4. If authors are going to use cumulative STI at 12-months as their primary outcome, they should state based on their pilot work how many participants would be needed to have at least 80% power to show a difference. We have included the sample size calculation for the main trial to pg. 10 Implications for the main trial: The sample size of the main trial is 5,000 participants. Of the pilot trial participants with an infection at baseline, 20% (18/89) had a positive chlamydia test at 12-month follow-up. Based on this and previous studies (3), we estimated the event rate (cumulative incidence of STI at 12 months) to be 20%. The trial is designed to detect a reduction in STI from 20% to 16% (RR 0.8). Taking into account the 2% contamination in the pilot trial, 5,000 participants are required to detect this reduction with 90% power, allowing for 20% loss to follow-up. 5. It would seem to me that given current ability to tailor and personalize computerized interventions, that the authors should have at least discussed the consideration of alternative texting designs or communication technology platforms that could improve effect sizes. The same could be said of content of messages, i.e. what messages seemed to resonate with participants. We have addressed this comment by adding an additional paragraph on pg. 10 Implications for the main trial: Modern mobile phones offer an array of options in which to deliver intervention content. For example, content can be delivered through app instant messaging, videos and audio and can involve bidirectional interactive content. The main trial will evaluate the content of the safetxt intervention messages that could be delivered by alternative modes in the future, such as instant messaging or private social media messaging, should the content be shown to be effective. Reviewer: 2 1. In the existing research section, it would be helpful to know the general functions or content of the messages from the other studies, for example, did they incorporate reminders, educational information, etc.? This information is only included for the third of the three studies cited. We have added additional information about the content of the interventions in this section (pg. 4 Existing research). 2. Also in this section, citation 13 noting for the statement that results of mobile phone support for safer sex behaviors is equivocal appears to be a review for all health behaviors, not just sexual behaviors. Then the three studies that follow suggest the results of prior research are not equivocal. Clarify briefly, or include some studies with null results.

6 We have amended this section to say that there is some evidence for mobile phone support on safer sex behaviours and have added referenced a newly published review, which emphasises the need for high quality trials (pg. 4, Existing research). 3. It is unclear where participants were recruited, until later in the methods. Consider moving the recruitment section earlier in the methods, perhaps before treatment and control group description or before outcomes section, or before eligibility criteria (and maybe moving outcomes). We have move the Eligibility criteria and Recruitment sections to before the Treatment groups section (pg. 4). 4. In the safetxt intervention section, it would be helpful to know the 12 behavior change techniques included in the intervention, or at least the higher order techniques/functions from the taxonomy. If the authors are concerned about word limits, there is some redundancy between the second and fourth sentence that could be consolidated. In addition, it is not stated how many and how frequently messages were sent (this information is only provided for the control group). We have consolidated the two redundant sentences and clarified that Supplementary file 1 provides information on both the intervention and control message frequency. (As in the response to Reviewer 1 comment 2): We have also included the intervention functions and added additional details of the intervention (pg. 5 the safetxt intervention). 5. Why was acceptability of intervention only assessed at 1-month follow-up instead of also at 3 and 12 months? One of the broader concerns in this field of mobile tech support is how sustainable participation can be, which can be reflected in opinions on acceptability of messages over time (in addition to the studies stated primary outcome of follow-up retention). We gathered data on acceptability at 1-month because the majority of messages were delivered within this period and we wanted to maximise participant recall. We have added this explanation on pg. 5 Acceptability of the intervention. 6. For recruitment procedures it could be helpful to briefly note that the proportion of clients who enrolled out of those contacted is presented in the results? We added a statement indicating that we recorded this data (pg. 5 Recruitment): We recorded the number of participants that were assessed, were eligible and declined. 7. The Masking section mentions in order to monitor the incoming text-messages, which implies that the intervention was bi-directional? Please clarify and describe the bi-directional functions if applicable. The Masking section seems overly detailed to this reviewer, if word limits are a concern. The intervention was not bi-directional, but participants could send text messages to our system. Participants did send occasional messages, mainly in response to requests for follow-up. We have corrected this sentence so that it does not imply that the intervention was bi-directional (pg. 6

7 Masking). 8. The Data collection and follow-up procedures section needs some attention to clarity. Why was follow-up response not assessed at 3 months? Both face-to-face and telephone recruitment were used, but it is unclear why and how? All participants enrolled by telephone had a positive test result. Some rationale or further explanation is needed. Regarding follow-up at 3 months, we have added numbers returning a chlamydia test sample to the second sentence on pg. 6 Data collection and follow-up procedures. Regarding mode of recruitment, we have added additional details in paragraph 2 pg. 6 Data collection and follow-up procedures: We recruited participants face-to-face in clinics and by telephone, depending on the service. At the Cambridge service, staff assessed eligibility when providing positive chlamydia test results over the phone. With the potential participant s agreement, staff at the service provided their telephone number to the Project coordinator, who then contacted the participant to enrol them and collect baseline data over the phone. 9. Table 1, I think the variable domain titles should be placed at the top of each section rather than at the bottom. We have reformatted this table (pg. 7 Table 1). 10. The variables on last time participant had sex with someone new the participant/partner was tested prior to sex need some clarification details, perhaps in the text or maybe a footnote. What was the timeframe on these questions? Was there any clarifying language included? These could be confusing to respond to from this reviewer s perspective. We have added a footnote and clarified these variables (pg. 7 & 8 Table 1): Testing prior to last sex with someone new and Partner testing at last sex with someone new* * Participants were asked The last time you had sex with someone new, did they get tested for sexually transmitted infections before you had sex? 11. The primary outcomes section needs a bit of explanation in the text. I think maybe inserting assessment of between for and cumulative incidence of chlamydia would suffice. We have amended this sentence accordingly (pg. 5 Outcomes, primary): The co-primary outcomes were the recruitment rate and the completeness of follow-up for assessment of the proposed primary outcome for the main trial, cumulative incidence of chlamydia at 12 months 12. Table 3 could be reformatted to make comparisons between 1 month and 12 month data easier, specifically, by making four columns for the data (control and intervention at 1 month and 12 month). Also, I don t think the data on The participant read the messages we sent to the other person and

8 how many the participant read is needed in the table since the data is sparse, the latter only applied to control group, and it is sufficiently described in the text. We have changed Table 3 on pg. 8 by removing the data on knowledge of anyone else in the study and sharing the messages between them as the point of this information is to assess potential for contamination and this is stated in the text in the last sentence of the Process outcome data section on pg. 8. We have also reported only intervention participant process data. 13. It is not clear to this reviewer that the study was not powered for behavioral outcomes such as condom use or even treatment completion (I agree that the STI incidence would require much larger sample in the context of treatment settings). The sample size is modestly large for a pilot. The pilot protocol states that one of the aims is to assess behavioral outcomes. The supplementary table (file 4) is not set up to make comparisons over time only comparisons between intervention and control are presented for each time point. Comparisons should be presented for Month 1 and 12 by intervention and control (and baseline ideally, was there a baseline?). It seems that there could be some statistically significant differences by intervention group over time for condom use at last sex (control had declined rate over time, intervention had increased rate over time, but there is the attrition issue to contend with). It is fine that this paper focuses on acceptability and process outcomes, but the behavioral outcome data in the supplementary files is distracting and not setup correctly to assess whether or not there were behavioral impacts, and in what direction those could have been. I think it would be better to either exclude the sexual behavior results completely (even from supplementary files), or do the analysis appropriately and state clearly that there were no significant intervention effects, then focus the limitation section on why that might be the case (and also include the supplementary file information). All the content on masking and blinding in the limitations is much less relevant than the implications of sample size, statistical power, and assessment methods for behavioral outcomes. (As in the response to Reviewer 1 comment 3): We have removed the report of the behavioural outcomes from this paper. We had discussed this previously, and in further discussion considering both reviewer s comments, we agreed that it s best not to include the behavioural outcomes. We appreciate that many trials do look at change in behaviour, however this was not how we designed this pilot study. The safetxt intervention aims to reduce STI through changing 3 key behaviours: condom use, testing and correct treatment (including partner notification). For partner notification, looking at change in behaviour would not have been possible. For testing, it did not seem appropriate for this population as they were all seeking a test at the outset. We could have looked at change in condom use over time, but we designed the trial to look at the behaviours between groups and in a similar way across the 3 behaviours. We acknowledge that looking at change in condom use over time may have given us a little more power to explore changes in this behaviour. We expect a messaging intervention to have at best modest benefits so our main point remains that this study was not powered to look at behavioural outcomes. 14. Comparison with other studies section is okay, but the interpretation is compromised by not including information on incentives (a key factor in participation and retention rates). Large scale surveys very often have lower response rates than smaller scale behavioral intervention trials. In addition, it is not clear how the rates presented for the other studies related to the initial enrollment rate of this study. Also, the first and last sentences of this section seem to contradict each other (i.e., higher follow-up vs. compares well with similar studies ). We have removed the comparison of our trial follow-up to survey follow-up and therefore the last

9 sentence as well (pg. 10 Comparison with other studies). 15. What additional follow-up strategies will be developed or are being considered or can be suggested? Since this is a primary topic of this paper, it would seem more relevant to discuss this topic than to simply end on more will be developed. We have addressed this on pg. 10 Comparison with other studies: Additional strategies to increase follow-up will be developed for the main trial, such as providing incentives to recruitment sites that have the highest proportion of participants completing follow-up. VERSION 2 REVIEW REVIEWER REVIEW RETURNED Dallas Swendeman Department of Psychiatry and Biobehavioral Sciences David Geffen School of Medicine University of California, Los Angeles United States of America 06-Sep-2016 GENERAL COMMENTS The revision is mostly responsive to both reviewers' comments. Relatively minor comments follow. 1. The intervention description should provide one sentence about the timing of the messages in the text (and not simply refer to supplementary material or provide the table in the text). Later in the manuscript it states that the process data was collected at 1- month follow-up because most of the messages were delivered in the first month, and then refers reader to supplementary material. This should be at least minimally descriptive in the text so readers do not have to refer to details in supplementary materials. 2. Limitations should address attrition and interpretation of the 12- month follow up process data. The percentages out of completers was presented in table 3 but those percentages could be much lower if one assumes the loss to 12-month follow-up participants did not find the intervention acceptable. 3. Process outcome data states that 97% of messages were delivered by our system and then Discussion states that 97% of messages sent were successfully delivered to participants phone. I m not sure the latter statement can be stated reliably. If so, readers will be very interested to know how delivery or receipt were confirmed for text-messages because this typically can not be done reliably by mobile phone service providers. I think being sent by a system is fine, it is the receipt or implied receipt ( delivered to phones ) that is questionable. 4. Discussion notes that the main trial will include mediation analysis of hypothesized mechanisms of action of the intervention. This statement demands some examples, or it should be cut from the discussion. 5. Finally, there should be a brief discussion of the non-mobile aspects of the intervention or research design that might have an intervention effect. Specifically, are home test kits and monetary

10 incentives for returning them part of a standard of care? I assume not but this should be stated either way. Also, what are the implications for replication in practice? Notably, the 12-month followup rates on test-kit returns were similar for intervention and control groups. Re-testing among those at risk is an important proximal intervention outcome in its own right. The main trial will allow inferences on the impact of text-messaging intervention versus none, but it might be unclear whether or not the text-messaging intervention would work in the absence of home tests and incentives. VERSION 2 AUTHOR RESPONSE The intervention description should provide one sentence about the timing of the messages in the text (and not simply refer to supplementary material or provide the table in the text). Later in the manuscript it states that the process data was collected at 1-month follow-up because most of the messages were delivered in the first month, and then refers reader to supplementary material. This should be at least minimally descriptive in the text so readers do not have to refer to details in supplementary materials. We have added the following sentence (p5 The safetxt intervention): Around half of the messages in each set are delivered in the first month: 59% (37/63) of the womenpositive, 59% (36/61) of the men-positive, 49% (25/51) of the women-negative and 49% (24/49) of the men-negative messages (Supplementary file 1). We have also changed the majority to around half to the following sentence for accuracy (p5 Acceptability of the intervention): We gathered participant views at 1-month because around half of the messages were delivered within this period and we wanted to maximise recall. 2. Limitations should address attrition and interpretation of the 12-month follow up process data. The percentages out of completers was presented in table 3 but those percentages could be much lower if one assumes the loss to 12-month follow-up participants did not find the intervention acceptable. We have added the following to p9 Strengths and limitations: We present a complete case analysis of the process outcomes and participants views about the acceptability of the intervention. The main trial will include sensitivity analyses in relation to missing data, which could reveal that participants lost to follow-up were less likely to read the messages and less likely to find them acceptable. 3. Process outcome data states that 97% of messages were delivered by our system and then Discussion states that 97% of messages sent were successfully delivered to participants phone. I m not sure the latter statement can be stated reliably. If so, readers will be very interested to know how delivery or receipt were confirmed for text-messages because this typically cannot be done reliably by mobile phone service providers. I think being sent by a system is fine, it is the receipt or implied receipt ( delivered to phones ) that is questionable. Our system receives a confirmation for each successful message delivery. In instances where the number is invalid, the device is overseas or if the phone is in an area without network connectivity, the message may not be delivered. In this case, the gateway attempts to deliver the message

11 approximately 5 times. After these attempts, our system is notified that the message was not delivered. We have clarified this and amended the sentence on p8 Process outcome data so that it is consistent with the sentence on p9 Principal results: All intervention messages were sent successfully from our system. Over 97% of the intervention messages were successfully delivered to participants phones, as confirmed by the gateway. 4. Discussion notes that the main trial will include mediation analysis of hypothesized mechanisms of action of the intervention. This statement demands some examples, or it should be cut from the discussion. We have added a few examples on p10 Implications for the main trial: The main trial will include a mediation analysis of the hypothesised mechanisms of action of the intervention, for example, we will investigate if the intervention increases condom use self-efficacy and self-efficacy in telling a partner about an infection. 5. Finally, there should be a brief discussion of the non-mobile aspects of the intervention or research design that might have an intervention effect. Specifically, are home test kits and monetary incentives for returning them part of a standard of care? I assume not but this should be stated either way. Also, what are the implications for replication in practice? Notably, the 12-month follow-up rates on test-kit returns were similar for intervention and control groups. Re-testing among those at risk is an important proximal intervention outcome in its own right. The main trial will allow inferences on the impact of text-messaging intervention versus none, but it might be unclear whether or not the textmessaging intervention would work in the absence of home tests and incentives. On p10 Implications for the main trial we have added: The postal test kits and monetary incentives for returning them were not part of usual care during the pilot trial, but part of the pilot trial procedures and delivered to participants in both the intervention and the control group. In the main trial we are sending postal test kits at 12-months only, with participants unaware of the incentive until they receive the kit. It is possible that offering incentives at 12-months could influence self-reported behavioural outcomes, but this would be equal in both groups. The incentive could not influence whether or not participants had an STI at 12-months. VERSION 3 REVIEW REVIEWER REVIEW RETURNED Dallas Swendeman University of California, Los Angeles United States of America 15-Oct-2016 GENERAL COMMENTS The authors have addressed the reviewers comments satisfactorily.