Commentary to Pape HENRIK NATVIG Primary prevention of alcohol and drug use in junior high school Suspicious scientists, lack of ecological validity and general pessimism Pape s critical literature review of whether evaluations hold adequate quality focuses on two problems: 1) Too many papers presenting the program implementers intended effects get published while negative results are often underreported. This is partly due to the scientists wishes to highlight the positive results in the programs they have contributed to and have economic interest in, and partly due to a bias in which negative results are difficult to get published. 2) Ecological validity can the positive results from intervention programs researchers implement be repeated, when ordinary teachers are left alone to implement the programs on a large scale? While one easily agrees that negative or a lack of results are underreported, the reasons are rather less suspicious than those suggested by Pape (pp. 524 525). Authors and referees evaluating such papers will suspect that something went wrong in the research process. Unidentified causes of negative results easily render the paper to be unworthy to be published. The opposite is also true - theoretically sound intended effects are reported, as one believes that the study A shorter version of my Norwegian commentary to Hilde Pape s Norwegian paper Skoletiltak som synes å gi effekt: Fruktbar forskning om rusforebygging eller suspekte historier om suksess? Nordisk alcohol- & narkotika tidskrift 26 (5) 341 364. 536 NORDIC STUDIES ON ALCOHOL AND DRUGS VOL. 26. 2009. 6
documents true connections from the real world. I miss a comprehensive literature review showing the usefulness of primary prevention, for instance from adherents of Botvin s LST or Project ALERT, or from other health-related behavioral fields like tobacco (Vær RøykFRI). In this way, the belief in the possibility to develop useful programs could gain support. Pape et al. (2006) follow Room (2005) and seems to recommend that intervention programs should be limited to provide alcohol- and drug-related harm information, thereby enhancing popular support for structural interventions (closing times, prices, age limits etc.). Ecological validity is important programs also need to have behavioral effects when teachers take over the responsibility from program developers. However, in my opinion, Pape is too critical and throws the baby out with the bath water. The fact that schools fail to implement a program correctly does not mean that the program is useless or cannot be recommended. Its failure might be due to poor motivation and knowledge among the implementers and not to the program itself. This would be similar to claiming that antibiotics are ineffective simply because many people in developing countries fail to use them as prescribed. The implementation might be the problem not the drug and the intervention program. Of course, improvements in alcohol- and drug-related prevention programs are needed. Youth & Alcohol evaluation Pape s description of the reseach group s report (Nordahl et al. 2006) gives the wrong impression. The program qualified as category 3, which meant it documented results according to a list of specifications we gave on page 10 in the report (science based, positive behavioral effect etc.). Pape (pp. 522 and 529; Pape et al. 2007) misinterpreted our text category 3 and our descriptions were not recommendations for schools to use the program, it simply fitted within the specifications. It was not in our mandate to evaluate whether the program should be implemented in schools or not. Since we now know it could be misunderstood, I regret we did not state even clearer that the unstructured version did not have documented behavioral effects. However, it is misleading (Pape p. 522; Pape et al. 2007) to say we (Nordahl et al. 2006) concealed a negative effect. The unstructured version does not seem to have negative effects either more of a lack of effect (se below). Both the Addiction paper (Wilhelmsen et al. 1994) and her thesis (Wilhelmsen 1997) left us with the impression of high scientific quality compared to the rest of the 15 Norwegian programs. Most lacked methodically good evaluations. We explicitly connected the positive results to the intervention group following the structured version: The evaluation shows positive results in the perception of alcohol and in alcohol use in the group following the most structured version. (Nordahl et al. 2006, 109; my translation). Under the unfortunate heading Recommendations (the same heading for all 16 programs), we further specified that the program had to be implemented in accordance with the guidelines for the structured version. Empirical support for the effect of the structured version were documented and it was beyond our mandate to go into further details in the program, including examin- NORDIC STUDIES ON ALCOHOL AND DRUGS V O L. 26. 2009. 6 537
ing what consequences it might have to implement it in an unstructured manner. Finally, we clearly stated that there may be a need for a more thorough quality control of the program (Nordahl et al. 2006, 109). Central to Pape s criticism is Wilhelmsen et al. s (1994) alcohol measure which, according to Pape (note 1 p. 531), is unfit to show change or stability over time. Yes, the sum score is illogical and has poor content validity; however, it functions well among 13 15 year-olds. Criteria validity can equally well be used to validate an operationalization. Different types of alcohol scores have been checked in several datasets of 13 15 year-olds including the logical frequency x quantity (see Natvig Aas 1995). Criteria validity showed the best operationalization to be the illogical sum score of frequencies of drinking beer, wine and spirits. This sum score correlated better with alcohol expectancies, alcohol related self-efficacy measures and other indicators (Aas 1993; Aas et al. 1995). It also ranked the adolescents better longitudinally (higher correlation in drinking over time). Frequency is closely correlated to quantity, r=.81 among 12 19 year-olds (Aas & Klepp 1992). Based on the criteria validity, the inclusion of quantity impaired the measure. Correlations were lower - perhaps because the question of average quantity might be too difficult for 13-15 year-olds. I therefore defend the use of different criteria to validate an operationalization of alcohol use compared to content validity. These criteria are theoretically associated with alcohol use and they correlate better (in different samples). In a sample of over 2,000 13 and 15 yearolds (Aas et al. 1995), the intention to drink correlated at.69 with a very similar alcohol sum score as that of Wilhelmsen et al. (1994). New data analyzes With permission from Wilhelmsen, I recreated a data file from her raw data. Wilhelmsen et al. (1994) split the sample (N=855) into three groups: LS=Little Structure, HS=High Structure, and a control group. The alcohol sum score was the sum of frequency of drinking beer, wine and spirits all scored from 0 to 3. Extreme scores (n=8 at T1 and n=24 at T2) were recoded to 5. Because I had to do my own missing replacements and excluded those who did not respond at both time points (the original paper had 8 more adolescents at T1 than T2) the figures differ slightly. Between week 2 and 15 (Wilhelmsen 1997), alcohol use is expected to increase among 13 year-olds. Therefore, stability might be considered a positive effect. A negative effect would be evident if the intervention group s drinking increased more than expected (than in the control group). In addition to Wilhelmsen et al. s (1994) sum scores, Table 1 also presents a new variable, a self reported frequency of being inebriated/drunk [beruset] during the last 6 months which has never been published to my knowledge. Response categories were zero (0); once (1); 2-4 times (2); 5 10 (3); and over 10 (4). At T1 and T2, 91.5% and 87.7% had never been drunk, an increase of 30.9% who reported to have been drunk during the 13-week period. Only the HS-group lowered their alcohol sum score (.71 to.55) - the two other groups increased. Although the changes are small, it is surprising to find a decrease in this age group at all. The increase in the LS-group (.81 to.96) looks rather substan- 538 NORDIC STUDIES ON ALCOHOL AND DRUGS V O L. 2 6. 2 0 0 9. 6
Table 1. Mean alcohol use sum scores and inebriation scores for the two intervention groups and the control group (Wilhelmsen et al. s (1994) figures in parenthesis). Alcohol sum score Inebriation score N T1 T2 T1 T2 Little structure 320 (314).81 (.80).96 (.90).18.31 High structure 273 (279).71 (.71).55 (.53).11.15 Control group 259 (262).69 (.67).73 (.69).14.24 Total 852 (855).74.76.15.23 tial compared to the control group (.69 to.73) (even bigger in my recreated figures). This suggests that the LS version can increase alcohol use. However, the changes in the inebriation score suggest that the LS and control group are very similar (.18 to.31 and.14 to.24). Even though the HSgroup increased from.11 to.15, this is still consistent with a desired behavioral effect, as the increase is smaller than the control group s. Correlations between alcohol expectancies (AEQ-A social scale) on one hand and a) alcohol sum score and b) inebriation score on the other (N=852): T1: a).55 and b).47; T2: a).62 and b).53 support Wilhelmsen s alcohol use operationalization. Mostly because of more variance in drinking, higher correlations among the older subjects are expected. The similarities in the correlations with AEQ-A suggest that Pape s criticism against the validity of the alcohol sum score is not important. The alcohol sums core s validity in Wilhelmsen s study is also supported by the pattern in inebriation scores, thereby supporting category 3 to Youth & Alcohol in the Nordahl report. Developing primary prevention programs Potential mediators are important for developing a more efficient prevention program. Hansen and McNeal s (1996, 502) law of maximum expected potential effect specifies that the magnitude of behavioral change a program can produce is directly limited by the strength of relationships between mediators and targeted behaviors. In other words, we should focus on the predictors that correlate strongest with the behavior. Components from Ajzen s (1988) TPB is a minimum (attitudes, injunctive norms, and perceived behavioral control) but also self-image, values, commitment, anticipated regret, experience in the situation, descriptive and moral norms, expectancies and implementation intentions should be among the predictors to consider. To further develop prevention programs, we need many new research-based interventions as recommended by Hansen and McNeal (1996), which should be repeatedly redesigned based on sound evaluations. Pape argues that the results so far are so poor that it is a waste of resources to implement prevention programs. To me, this sounds like throwing the baby NORDIC STUDIES ON ALCOHOL AND DRUGS V O L. 26. 2009. 6 539
out with the bathing water. Pape gives the impression that because the interventions are of poor quality, they should be replaced with information campaigns. In my opinion, we are only in the first days of developing science-based prevention programs. I agree with Botvin (2000) who recommends prevention programs aimed at early teenagers that teach drug resistance skills and norm setting either alone or in combination with general personal and social skills. Hansen and McNeal (1996) provide guidelines for developing better prevention programs. There is a vast need for research-based prevention programs in Norway. Even in the USA, it is only in recent years that research-based programs are becoming widespread. In 2005, Ringwalt et al. (2009) examined 1,721 schools in the USA and found that only 42.6% of schools with prevention programs used research-based programs. However, this was an eight percentage point increase since 1999. The USA s two most important research-based programs were Botvin s LST and Project ALERT, which were used by 19% of the schools. Henrik Natvig, Dr. Psychol. Department of Psychology University of Oslo P.O. Box 1094 Blindern N-0317 Oslo, Norway E-mail: henrik.natvig@psykologi.uio.no REFERENCES Aas, H. (1993): Adaptation of the Alcohol Expectancy Questionnaire (AEQ-A): A short version for use among 13-year-olds in Norway. Scandinavian Journal of Psychology 34: 107 118 Aas, H. & Klepp, K.-I. (1992): Adolescents alcohol use related to perceived norms. Scandinavian Journal of Psychology 33: 315 325 Aas, H. & Klepp, K.-I. & Laberg, J. C. & Aarø, L. E. (1995): Predicting adolescents intention to drink alcohol: Outcome expectancies and self-efficacy. Journal of Studies on Alcohol 56: 293 299 Ajzen, I. (1988): Attitudes, personality, and behavior. Belmont, Calif.: Dorsey Press Botvin, G. J. (2000): Preventing drug abuse in schools: Social and competence enhancement approaches targeting individual-level etiological factors. Addictive Behaviors 25: 887 897 Hansen, W. B. & McNeal, R. B. Jr. (1996): The law of maximum expected potential effect: constraints placed on program effectiveness by mediator relationships. Health Education Research, Theory & Practice 11: 501 507 Natvig Aas, H. (1995): Alcohol expectancies and socialization: Adolescents learning to drink. Doctoral thesis, HEMIL-centre, Psychological Faculty, University of Bergen Nordahl, T. & Natvig, H. & Samdal, O. & Tyholdt, R. & Wilhelmsen, B. U. (2006): Vurdering av rusprogrammer i skolen. In: Nordahl, T. & Gravrok, Ø. & Knudsmoen, H. & Larsen, T. M. B. & Rørnes, K. (eds.): Forebyggende innsatser i skolen. Oslo: Utdanningsdirektoratet Pape, H. & Baklien, B. & Rossow, I. (2007): Villedende om rusforebygging i skolen. Tidsskrift for Den norske legeforening 127: 1399 Pape, H. & Storvoll, E. E. & Rossow, I. (2006): Så feil kan man ta! En studie av unge menneskers kunnskap om alkohol- og narkotikabruk. Tidsskrift for Ungdomsforskning 6: 97 109 Ringwalt, C. & Vincus, A. & Hanley, S. & Ennett, S. & Bowling, J. & Rohrbach, L. (2009): 540 NORDIC STUDIES ON ALCOHOL AND DRUGS V O L. 2 6. 2 0 0 9. 6
The prevalence of evidence-based drug use prevention curricula in U.S. middle schools in 2005. Prevention Science 10: 33 40 Room, R. (2005): What to expect from a social aspects organization, and what to expect from school based alcohol education. Addiction 100: 1072 1073 Wilhelmsen, B. U. (1997): Development and evaluation of two educational programmes designed to prevent alcohol use among adolescents. Doctoral thesis, HEMIL-centre, Psychological Faculty, University of Bergen Wilhelmsen, B. U. & Laberg, J. C. & Klepp, K.-I. (1994): Evaluation of two student and teacher involved alcohol prevention programmes. Addiction 89: 1157 1165. NORDIC STUDIES ON ALCOHOL AND DRUGS V O L. 26. 2009. 6 541