Evaluating the effectiveness of health promotion policy: changes in the law on drinking and driving in California

Transcription

1 Health Promotion International, Vol. 21 No. 2 doi: /heapro/dak005 Advance access publication 7 April 2006 Ó The Author (2006). Published by Oxford University Press. All rights reserved. For Permissions, please journals.permissions@oxfordjournals.org Evaluating the effectiveness of health promotion policy: changes in the law on drinking and driving in California STEFANO CAMPOSTRINI 1, DEBORAH HOLTZMAN 2, DAVID V. MCQUEEN 2 and ELISA BOARETTO 3 1 Department of Applied Statistics and Economics, University of Pavia, Italy, 2 Centers for Disease Control and Prevention, Atlanta, GA, USA and 3 emme & erre, Management and Research, Padua, Italy SUMMARY The purpose of the study was to determine the utility of general population health surveillance data for evaluating broad policy changes that relate to health promotion. Data were drawn from the United States (US) Behavioral Risk Factor Surveillance System (BRFSS) for one US state, California. Because these data are collected frequently and continually, a quasi-experimental approach to the evaluation was possible using a type of interrupted time series analysis or longitudinal impact analysis. A statistically significant decrease in the number of declared episodes of drinking and driving was found after enactment of new state policy. These findings were compared and found consistent with another study in California that examined the effect of changes in the law on alcoholrelated traffic accidents. Our findings suggest that data from a behavioral surveillance system, in this case the BRFSS, are useful to evaluate the effect of a health promotion intervention. Further, the study demonstrates the utility of comparing different data sources when assessing a population-wide change in health promotion policy. Key words: evaluting health promotion policy; surveillance; risk factors; drinking and driving INTRODUCTION This article evaluates a complex phenomenon, namely how broad changes in policy at the population level can be monitored in order to determine whether such policies are effective. The broader ramifications of such an evaluation, in light of recent attention to the production of evidence for the effectiveness of health promotion interventions, are also addressed. Data collection systems that assess population morbidity and mortality, and more recently, systems that assess lifestyle or behavioral factors that contribute to morbidity and mortality are important in many ways. Such systems that collect data frequently and continuously are especially useful for analyzing the dynamic features of population health. The ideal model to monitor human behavior is a regularly repeated time series of cross-sectional surveys. However, it is quite clear that an actual continuous observation of the same individuals over time cannot be performed to assess human attitudes, opinions and behavior. Rather, a continuous data collection is defined as the product of a sequence of repeated surveys, or more precisely, it is characterized by multiple independent samples drawn sequentially over time, with observations of the same population not involving the same subjects. Further, the observations must be sufficiently near one another over time, avoiding gaps of such a consistency that the comprehension of what has happened between two observations would be uninterpretable. Data produced in this way can 130

2 Effectiveness of health promotion policy 131 be used to estimate the prevalence of healthrelated behaviors and practices, identify sociodemographic variations, provide information to target health programs and services, address critical health issues in a timely manner, guide legislation and policy, measure health objectives, and assess the impact of policies and programs at various levels on the population. One such data system is the US Behavioral Risk Factor Surveillance System (BRFSS). Our purpose here is to assess the usefulness of this data collection system for evaluating the impact of a particular state policy. In the United States, driving under the influence of alcohol (DUI) remains a major cause of crashes and injuries, particularly among people under 34 years of age (National Center for Health Statistics, 1994). For example, in 1994, 41% of the more than traffic fatalities were related to alcohol (National Highway Traffic Safety Administration, 1995). During the last decade of the 20th century, changes in the drinking and driving laws were introduced at the state level (Hingson et al., 1996), including two different changes introduced in California. The first, implemented in January 1990, changed the legal definition of DUI, lowering the allowable blood alcohol limit from 0.10 to 0.08%. The second, implemented 6 months later, immediately suspended the license of any person who violated the new limit. The purpose of the study, conducted by the Department of Statistical Science at the University of Padova, Italy, in collaboration with the US Centers for Disease Control and Prevention (CDC), was 2-fold, and used data from the BRFSS, a continuous survey carried out by all states in the United States, through the support and coordination of the CDC. We were interested foremost in the methodological issue of investigating whether data from a health surveillance system were useful for evaluating the effect of a policy intervention. Secondarily, we wanted to examine the substantive issue of assessing the impact of the changes in the law on drinking and driving among California adults. METHODS The BRFSS is a unique, state-based surveillance system (McQueen, 1996; Holtzman, 2003), currently active in all 50 states, the District of Columbia, Puerto Rico, Guam and the Virgin Islands. The system was designed to gather information on health behaviors and preventive practices primarily related to chronic disease and injury. Every month, a representative sample of persons 18 years or older is selected for interview in each participating state and territory. Because the system has been operating for two decades, it approximates a continuous data stream. It is the availability of this continuous data stream that allows us to assess the potential of these data for policy or intervention evaluation. For our analysis, we used data from California for the 3 years before and the 3 years after the year in which the interventions took place: a total of about interviews and 84 points of observation (from January 1987 to December ). We examined monthly mean values of the responses to the following question: During the past month, how many times have you driven when you ve had perhaps too much to drink? The question wording remained the same over the 7-year period. Because the BRFSS was designed to collect data continuously and frequently, a quasiexperimental approach to the evaluation was used (Campbell, 1963; Campbell and Ross, 1968; Caporaso and Pelowski, 1971). This is the so-called interrupted time series analysis or longitudinal impact analysis, introduced by the work of Box and Tiao (Box and Tiao, 1975; Box and Jenkins, 1976; McDowall et al., 1980). We wanted to observe whether the changes in the law produced a shift in the time series or any other change in its evolution; a method previously employed in studies using BRFSS or other data (Glass, 1968; Tyler and Brown, 1968; Ross et al., 1970; Deutsch and Alt, 1977; Escobedo et al., 1992). The model considers: a stochastic component (usually an ARIMA model) that takes into account all the dependencies among the observations and an intervention component of which we want to evaluate the significance. Formally, the model is: Y ¼ N t þ I t where N t is the time series stochastic component and I t is the intervention component. 1 For 2 months, September and October 1987, the relevant questions for this study were not included in the BRFSS questionnaire in California.

3 132 S. Campostrini et al. Table 1: Parameters indicating the change in law effect on drinking and driving behavior for the three models tested (data: California BRFSS, , n = ) Parameter Value Standard error T-ratio P-value Y t =v 1 I t1 +N t v Y t =v 2 I t2 +N t v v Y t =N t +v 1 I t1 +v 2 I t2 v I t1 is equal to 1 from January I t2 is equal to 1 from July The second component will be equal to zero until the observation corresponds to the intervention, at which point it will assume a non-zero value. If this value is statistically different from zero, we can conclude that the intervention had a significant effect. The intervention component can be modeled according to three different combinations of effects that capture the duration of the effect (temporary or permanent) and the onset of the effect (gradual or immediate). Thus, we distinguish: an immediate and permanent impact; an immediate and temporary impact; a gradual and permanent impact. From a first graphic analysis of the time series considered, we tested models representing an immediate and permanent impact. This model was the most sensitive to the nature of the data and to the type of the interventions we were evaluating, representing a type of interrupted time series with slightly variable homoscedasticity, but principally stable over a long time period. Consequently, the model we chose was: Y t ¼ vi t þ N t where N t is an ARIMA (p, d, q) 2 to be identified, I t is a step function assuming a value of 1 from the observation corresponding to the intervention or 0 otherwise and v is the parameter measuring the effect on the observed series. 2 Three parameters characterize an ARIMA model: (p) the AutoRegressive parameter (the process is considered as a linear function of the p preceding values); (d) the order of Integration (typically not more than 2 for modeling purposes; it is convenient to eliminate the non-stationarity by means of d differences of the original series); and (q) the Moving Averages parameter (a linear combination of q random processes). We tested three models (Table 1): the effect of the January 1990 intervention [reduction of the allowable blood alcohol concentration (BAC) from 0.10 to 0.08%]; the effect of the July 1990 intervention (license suspension); and the effect of the combination of both interventions. We also indirectly assessed validity by comparing our findings with those of another study conducted in California (Rogers, 1995). Using intervention time series analysis, Rogers evaluated the influence of the new DUI law among the general population of DUI offenders, as measured by the number of alcohol-related traffic accidents. The California Highway Patrol (CHP) provided the accident data from the Integrated Traffic Records System (SWIRTS); these data are not self-reported and in view of how they are collected are assumed to be valid. RESULTS In all three cases, neither autocorrelation nor trend was detected, indicating a white noise model. Of the three, the third was found to be the best fit. Table 1 reports the numerical results. The first parameter (v 1 ), which considered the effect of the first intervention (BAC reduction), was positive, i.e. it seems that the introduction of the law has increased the number of people declaring episodes of drinking and driving in the month previous to the interview. A possible explanation for the positive value is that the new law did not change the actual number of declared episodes of drinking and driving, but instead influenced (to a lesser extent) only the number of people considering themselves to have drunk perhaps too much. However, the parameter was not significant (P > 0.2), indicating no statistically significant effect of the first law. The other parameter is higher in absolute value

4 Effectiveness of health promotion policy 133 0,30 0,25 0,20 Low ering of the BAC Limit License Suspension 0,15 0,10 0,05 0,00 Feb-87 Feb-88 Feb-89 Feb-90 Month Feb-91 Feb-92 Feb-93 Fig. 1: The time series: monthly average estimated number of declared episodes of drinking and driving per person, California BRFSS, , n = BAC = blood alcohol content. ð ^v 2 ¼ 0:08Þ and statistically significant. The real change occurred only after the second law was enacted, which produced a substantial decrease in the number of declared episodes of drinking and driving (Figure 1). Results of the CHP comparison were consistent with those from our study, in that both alcohol-related accidents and declared drinking and driving episodes decreased with the enactment of the second law, license suspension, but less so with the first law which lowered the BAC limit. Moreover, the analysis conducted on the time series of accidents (Rogers, 1995) with an identical approach to that in our analysis, led to the same conclusions: the first intervention did not have a statistically significant effect on the time series, while the second one produced a highly significant change. DISCUSSION Our findings suggest that changes in the drinking and driving law, specifically suspending the license of those who break the law, decreased drinking and driving behavior among adults in California. Our findings also suggest that data from a behavioral surveillance system, such as the BRFSS, can be useful to evaluate the effect of such a change. One often cited limitation of this type of survey is that it gathers selfreported information. Some behaviors, however, are not easily observable and self-reported information may be the only option. Furthermore, over or under estimates of behaviors that may result from self-reports would not influence our findings with respect to trends. Quite often in public health, the primary interest is on whether and how change occurs, i.e. is there an increasing or decreasing trend for a particular phenomenon; generally there is less interest in the actual value of a specific variable. For example, knowing that the percentage of smokers among an adolescent population over a certain time period is increasing may be much more useful than having a precise estimate of that percentage. The methodological consequence is that less attention can be paid to biases in the estimated value of a variable, since, if we can assume that these are stable over time (as quite often is reasonable in repeated measurements based on independent samples), the possible bias will not influence trend estimates. In addition, a number of studies have shown that estimates from BRFSS for selected healthrelated behaviors are similar to those based on in-person interviews or observations (Gentry et al., 1985; Anda et al., 1989; Nelson, 1996; Nelson et al., 2001). We were also able to compare our results with another California study that carried out analyses using actual data from

5 134 S. Campostrini et al. alcohol-related traffic accidents, rather than data from self-reports. The similarity of our results with this study lends support to the validity of our conclusions. More generally, we believe that in many cases it is possible to use behavioral surveillance data for evaluation purposes; the study here is one example. In these analyses, of course, validity issues must be always addressed, but quite often data for analysis of an intervention s effect are difficult or impossible to collect, for ethical, feasibility or cost reasons. Furthermore, when ad hoc data are collected, selection bias, attrition and other problems intervene, severely jeopardizing the reliability and the validity of the results. Therefore, the continuous data stream offered by surveillance systems, such as the BRFSS, are often a reasonable alternative, certainly much less demanding of resources and sometimes as valid, or even more so. Most importantly, our approach addresses the fundamental challenge for health promotion of how to show, in a scientifically acceptable way that a policy change, stemming from a health promoting ideology, can be evaluated. Furthermore, it addresses the issue of complexity which is always present in assessing such broad population effects. Finally, it illustrates that we must have and use institutionalized systems of monitoring (Campostrini and McQueen, 2005) if we are to attribute cause to large scale policy changes that only manifest their effects over longer periods of time. Many of the challenges related to evaluating policy have been addressed in recent years (Campostrini, 2003). Conversely, McQueen and Springett (McQueen and Springett, 2001) have shown, reflecting many of the discussions of the WHO European Working Group on Health Promotion Evaluation, that there are a dearth of ready strategies to evaluate the effectiveness of health promotion policies at the population level. This article provides one such approach. ACKNOWLEDGEMENT The authors would like to acknowledge the state of California BRFSS staff. Address for correspondence: Stefano Campostrini Department of Applied Statistics and Economics University of Pavia Corso Strada Nuova, Pavia Italy stefano.campostrini@unipv.it REFERENCES Anda, R. F., Dodson, D. L. and Williamson, D. F. (1989) Health promotion data for state health departments: telephone versus in-person survey estimates of smoking and alcohol use. American Journal of Health Promotion, 4, Box, G. E. P. and Jenkins, G. M. (1976) Time Series Analysis: Forecasting and Control. Holden-Day, Oakland, CA. Box, G. E. P. and Tiao, G. C. (1975) Intervention analysis with applications to economic and environmental problems. Journal of the American Statistical Association, 70, Campbell, D. T. (1963) From description to experimentation: interpreting trends as quasi-experiments. In Harris, C. W. (ed.), Problems of Measuring Change. University of Wisconsin Press, Madison, WI. Campbell, D. T. and Ross, H. L. (1968) The Connecticut crackdown on speeding: time series data in quasiexperimental analysis. Law and Society Review, 3, Campostrini, S. (2003) Surveillance systems and data analysis. In McQueen, D. V. and Puska, P. (eds) Global Behavioral Risk Factor Surveillance. Kluwer, New York, pp Campostrini, S. and McQueen, D. V. (2005) Institutionalization of social and behavioral risk factor surveillance as a learning system. Social and Preventive Medicine, 50, S9 S15. Caporaso, J. A. and Pelowski, A. L. (1971) Economic and political integration in Europe: a time series quasiexperimental analysis. American Political Science Review, 65, Deutsch, S. J. and Alt, F. B. (1977) The effect of Massachusetts gun control law on gun-related crimes in the city of Boston. Evaluation Quarterly, 1, Escobedo, L. G., Chorba, T. L., Remington, P. L., Anda, R. F., Sanderson, L. and Zaidi, A. A. (1992) The influence of safety belt laws on self-reported belt use in the United States. Accident Annuals and Prevention, 24, Gentry, E. M., Kalsbeek, W. D. and Hogelin, G. C. (1985) The behavioral risk factor surveys: II. Design, methods, and estimates from combined state data. American Journal of Preventive Medicine, 1, Glass, G. V. (1968) Analysis of data on the Connecticut speeding crackdown as a time series quasi-experiments. Law Society Review, 3, Hingson, R., Heeren, T. and Winter, M. (1996) Lowering state legal blood alcohol limits to 0.08%: the effect on fatal motor vehicle crashes. American Political Science Review, 86, Holtzman, D. (2003) The analysis and interpretation of data from the Behavioral Risk Factor Surveillance System (BRFSS). In McQueen, D. V. and Puska, P. (eds) Global Behavioral Risk Factor Surveillance. Kluwer, New York, pp McDowall, D., McCleary, R., Meidinger, E. E. and Hay Jr, R. A. (1980) Interrupted Time Series Analysis. Sage Publications, Beverly Hills, CA. McQueen, D. V. (1996) Surveillance of health behavior. Current Issues of Public Health, 2, McQueen, D. V. and Springett, J. (2001) Introduction to part 4: policies and systems. In Rootman, I., Goodstadt, M., Hyndman, B., McQueen, D., Potvin, L., Springett, J. and Ziglio, E. (eds) Evaluation in Health

6 Effectiveness of health promotion policy 135 Promotion: Principles and Perspectives. WHO Regional Publications, European Series, 92, pp National Center for Health Statistics (1994) Health, United States, National Center for Health Statistics, Hyattsville, MD. National Highway Traffic Safety Administration (1995) Traffic Safety Facts, US Department of Transportation, National Highway Traffic Safety Administration, Washington, DC, DOT HS Nelson, D. E. (1996) The validity of self-reported data on injury prevention behavior: lessons learned from observational and self-reported surveys of safety belt use in the United States. Injury Prevention, 2, 67. Nelson, D. E., Holtzman, D., Bolen, J., Stanwyck, C. and Mack, K. (2001) Reliability and validity of measures from the Behavioral Risk Factor Surveillance System (BRFSS). Social and Preventive Medicine, 46, Rogers, P. N. (1995) The general deterrent impact of California s 0.08% blood alcohol concentration limit and administrative per se license suspension laws: Volume 1 of an evaluation of the effectiveness of California s 0.08% blood alcohol concentration limit and administrative per se license suspension laws. (Report No. 158). Sacramento: California Department of Motor Vehicles. Ross, H. L., Campbell, D. L. and Glass, G. V. (1970) Determining the effects of a legal reform: the British breathalyzer crackdown of American Behavioral Scientist, 13, Tyler, V. O. and Brown, G. D. (1968) Token reinforcement of academic performance with institutionalized delinquent boys. Journal of Educational Psychology, 59,