USE OF CONSUMER PANEL SURVEY DATA FOR PUBLIC HEALTH COMMUNICATION PLANNING: AN EVALUATION OF SURVEY RESULTS. William E. Pollard

Transcription

1 USE OF CONSUMER PANEL SURVEY DATA FOR PUBLIC HEALTH COMMUNICATION PLANNING: AN EVALUATION OF SURVEY RESULTS William E. Pollard Office of Communication Centers for Disease Control and Prevention 1600 Clifton Rd., N.E., Mailstop D-42 Atlanta, Georgia KEY WORDS: Survey Methods, Mail Panel Surveys, Health Communication Data OVERVIEW The Office of Communication at the Centers for Disease Control and Prevention licenses syndicated market research data for understanding audiences in health communication planning. Market research data bases are widely used in the commercial sector to analyze audiences and develop messages to promote products and services to potential customers. They contain proprietary and public information on sociodemographic characteristics, consumer behavior, lifestyle activities, and media habits of potential customers, and are available, with individual identifying information removed, through licensing and contractual agreements. These data can be used to identify population segments to target with a message, and to identify audience segments that may differ in interests, lifestyle, and media habits in order to design messages with appropriate design, content, and media channels. A widely-used method for obtaining such data in the field of market research is through sample surveys where the samples are drawn from pools of individuals and households that are pre-recruited for use in various surveys, and these pools are known as omnibus or multi-purpose panels (Burns and Bush, 2000, Ch. 5). Because this method differs from national probability sampling methods typically used in health research, in this paper we examine the reliability and validity of results from panel surveys for use in health communication planning. Multi-purpose Consumer Panels in Market Research Panels are made up of groups of individuals and households who have agreed to participate in periodic surveys concerning products and lifestyle. Commercial marketing firms develop and maintain nationwide panels of several hundred thousand to more than a million individuals across the range of census demographics for use in market research. As discussed in a research report by one such firm (Market Facts, 1994), panels have several advantages for conducting surveys: (1) response rates are high; (2) attrition rates in longitudinal research are low; (3) customized samples can be easily constructed from data on hand; (4) samples can be made demographically representative on multiple variables from data on hand; (5) much respondent and household information is already on file which saves time and space on surveys; (6) they facilitate otherwise very difficult or expensive research, such as surveys of children and brand loyalty studies. However, the self-selection involved in agreeing to participate in a panel raises questions about the representativeness of the results. Are participants in panels different from the general population in their survey responses? In this paper, we address this question by comparing results from national mail panel surveys with those obtained from national probability sample surveys. PREVIOUS RESEARCH In the Market Facts (1994) report cited above, the researchers examined the comparability of results obtained from panel and non-panel samples by conducting parallel telephone surveys of a nationally representative random sample of members from their panel of over 600,000 U.S. households, and a national non-panel 2720

2 sample obtained through conventional random digit dial procedures. They found that the two samples were very similar on consumer behavior, lifestyle, and health status items. In addition, they found no differences in positive outlook, altruism, and amount of free time. In an extensive, widely-discussed study of civic involvement in the U.S., Putnam (2000) used results from the General Social Survey and the DDB Needham Lifestyles survey. The General Social Survey is a personal interview survey conducted by the National Opinion Research Center (NORC) using a national probability sample. The Lifestyles survey is a national mail panel survey conducted annually with independent samples by Market Facts for DDB Needham, a large advertising agency. The Lifestyles samples are drawn from the panel through stratified random sampling. The two surveys contained a number of similar opinion and lifestyle items. Putnam compared results of the surveys over a period of 25 years and found very close agreement in (1) the level of response, (2) trends over time, and (3) the pattern of demographic correlates. He writes that for purposes of explaining this wide range of attitudes and behavior, the two surveys are virtually indistinguishable (p. 422). Objectives CURRENT STUDY The objective of this study is to examine the reliability and validity of a health communication data base in which the data are obtained from a national mail panel survey. Reliability is assessed in terms of stability of item results over time: results should be stable from year to year in the absence of known trends or events that would be expected to cause changes. Validity is assessed in terms of agreement with other measures of the items that are themselves considered to be valid. Methods In this paper we examine data from the HealthStyles mail panel survey conducted for Porter Novelli, a social marketing and public relations firm. This survey has been conducted annually with independent samples since 1995 and contains items on health attitudes, behaviors, conditions, and knowledge. From 1995 to 2001, the survey has been administered by Market Facts to respondents to the DDB Needham Lifestyles panel survey, with oversampling for low income and minority groups. The sample size is around 2,500 to 3,000 respondents per year. Results are poststratified and weighted to US census benchmarks on age, race, sex, income, and family size (Korn and Graubard, 1999; Lohr, 2000). For a full description of the survey see Maibach, et al. (1996). A number of the items on the HealthStyles survey are the same as, or similar to, items on the Behavioral Risk Factor Surveillance System (BRFSS) survey. This is a national probability sample telephone interview survey. It is a collaborative project of the CDC and U.S. states. The sample size for core items administered in all interviews is 150,000 to 180,000 respondents annually. It is an accepted and widelyused public health surveillance and planning survey. Annual statistics for selected items from the survey are summarized on the CDC web site ( and are used here in the comparisons with HealthStyles items. Most of the national prevalence percentages reported in the tables on the web site are medians of state values. There were nine items from the BRFSS on the web site that were comparable to items on the HealthStyles survey from 1995 to These include: Health Conditions: arthritis, asthma, diabetes, high blood pressure, and overweight or obese, where this weight category is defined in terms of Body Mass Index (BMI) of or greater; Attitudes: health status self-rating; Behaviors: flu shot, pneumonia vaccination, smoke cigarettes. Note that not every item is measured every year on either survey. The HealthStyles survey is not a surveillance survey, but is designed to provide data for understanding audiences for communication and health promotion planning. Many of the items do not have counterparts on other surveys that can be used to examine validity. However to the extent that results for the above items can be shown to be valid in comparison with results from an accepted national probability sample survey, it provides indirect support for the validity of the results of other items on the survey. Results The basic findings are shown in the bar charts in Figure 1. The 95% confidence interval for the HealthStyles values is approximately plus or minus two percentage points around the reported value. In terms 2721

3 of reliability, the HealthStyles results are generally stable from year to year. The systematic changes over time that do appear are consistent with known trends mentioned below. In terms of validity, the levels are generally within a few percentage points of those from the BRFSS. Both surveys show trends for increasing prevalence of diabetes and overweight or obese (Mokdad, et al., 2000; Flegal, et al., 2002). In addition, further analyses showed similar demographic patterns in both data sets. For diabetes, both data sets showed increasing prevalence with increasing age of respondents, and higher prevalence in black respondents. For overweight or obese, both showed increasing prevalence with increasing age or respondents until age 65+, where prevalence declines, and higher prevalence in black respondents. Overall, there was agreement on levels, trends over time, and demographic breakdowns. In quantitatively summarizing the validity, for the nine items over the seven years, there were 34 same-year data pairs where the two surveys could be directly compared. For these 34 pairs of percentages, the average difference was 2.35 percentage points and the average of the absolute value of the differences was 2.88 percentage points. A scatterplot of the HealthStyles and BRFSS percentages is shown in Figure 2. For these 34 paired percentages, the correlation between the HealthStyles and BRFSS values was r =.99, reflecting the close agreement. CONCLUSION The findings provide support for the reliability and validity of the HealthStyles mail panel data for the items examined, and provide indirect support for the reliability and validity of other communication-relevant items on the survey. Data from the HealthStyles mail panel survey complement existing surveillance data and provide useful information for understanding audiences in health communication planning. These findings, along with those from the previous research, also have some more general practical and theoretical implications for survey research. As mentioned earlier, panels have a number of significant advantages for facilitating the conduct of surveys and reducing costs. It would be helpful to have some general guidelines concerning the conditions under which panel and non-panel surveys will yield comparable results. Additional empirical and theoretical work on this topic would be very useful to survey researchers. REFERENCES Burns, A.G. and Bush, R.F. (2000). Marketing Research, (3 rd. ed.). Upper Saddle River, NJ: Prentice Hall. Flegal, K.M., Carroll, M.D., Ogden, C.L., and Johnson C. (2002). Prevalence and trends in obesity among US adults, Journal of the American Medical Association, 288(14), Korn, E.L. and Graubard, B.I. (1999). Analysis of Health Surveys. NY: John Wiley & Sons, Inc. Lohr, S.L. (1999). Sampling: Design and Analysis. Pacific Grove, CA: Duxbury Press. Maibach, E., Maxfield, A., Ladin, K., and Slater, M. (1996). Translating health psychology into effective health communication: The American HealthStyles Audience Segmentation Project. Journal of Health Psychology, 1, Market Facts (1994). Mail Panels vs. General Samples: How Similar and How Different? Research on Research, Report No. 59. Arlington Heights, IL, Market Facts. Mokdad, A.H., Ford, E.S., Bowman, B.A., Nelson, D. E., Engelgau, M.M., Vinicor, F., and Marks, J.S. (2000). Diabetes trends in the U.S.: Diabetes Care, 23(9), Putnam, R.D. (2000). Bowling Alone: The Collapse and Revival of American Community. Appendix I: Measuring Social Change. New York: Simon and Schuster. The author would like to express his thanks to Dr. Edward Maibach and Dr. Deanne Weber from Porter Novelli for their encouragement of this project, and to Dr. Weber for help in understanding the HealthStyles methodology and the features of the data files. 2722

4 Figure 1. Bar charts showing HealthStyles and BRFSS results from 1995 to

5 Figure 2. Scatterplot of HealthStyles and BRFSS values (r =.99). 2724