KNOWLEDGEPANEL DESIGN SUMMARY

Transcription

1 KNOWLEDGEPANEL DESIGN SUMMARY This document was prepared at the effort and expense of GfK. No part of it may be circulated, copied, or reproduced for distribution without prior written consent.

2 KnowledgePanel Overview KnowledgePanel is the largest national, probability-based panel that provides the highest level of accuracy and sample representativeness available in online research for measurement of public opinion, attitudes, and behaviors. The panel was first developed in 1999 by Knowledge Networks, a GfK company, with panel members who are randomly selected, enabling results from the panel to statistically represent the US population with a consistently higher degree of accuracy than results obtainable from volunteer opt-in panels (for comparisons of results from probability versus non-probability methods, see Yeager et al., 2011). Because every sample unit has a known selection probability, KnowledgePanel is not susceptible to the professional respondent problem and other hazards of opt-in online panels based on convenience sampling. Unlike opt-in panels, individuals can become panelists only after being randomly selected; no one can just volunteer to be a member. Originally, panelists were selected using random-digit dialing (RDD) but address-based sampling (ABS) has been employed since ABS provides a statisticallyvalid sampling method with a published sample frame of residential addresses that covers approximately 97% of U.S. households, including households that: Have unlisted telephone numbers Do not have landline telephones Are cell phone only Do not have current Internet access Do not have devices to access the Internet Non-Internet households that are selected in the sample are provided a web-enabled computer and free Internet service so they can also participate as online panel members. GfK s KnowledgePanel recruitment methodology uses the same or similar quality standards as mandated by the Office of Management and Budget in the "List of Standards for Statistical Surveys," which indicates that Agencies must develop a survey design, including selecting samples using generally accepted statistical methods (e.g., probabilistic methods that can provide estimates of sampling error). KnowledgePanel consists of about 55,000 adult members (ages 18 and older). The highly diverse Hispanic population is represented in KnowledgePanel with members recruited in both English and Spanish, thereby representing different levels of language proficiency and acculturation. In addition to adult members, KnowledgePanel also includes approximately 3,000 teens (ages 13 to 17) whose parents or legal guardians, usually themselves panel members, have provided their consent. The scientific underpinnings of KnowledgePanel make it an excellent sampling platform for a broad range of economic, health, political, and social studies, as well as for policy and program evaluations. This versatility is reflected in the list of leading foundations, universities, government agencies, non-profit organizations, and prominent companies that rely on KnowledgePanel for their research projects. Consistent with our approach to sampling, the AAPOR Online Task Force recently recommended that researchers use probability-based sampling when measuring population characteristics using online surveys.1 Address-Based Sample (ABS) Recruitment Methodology When KnowledgePanel members were first recruited in 1999 using RDD, all households received WebTVs for answering surveys. By 2002, panelists with their own computers connected to the Internet could 1 The Online Task Force Report, 2010, of the American Association for Public Opinion Research, at 2

3 complete surveys without a WebTV and, beginning in 2009, non-internet households were provided Windows-based laptop computers and other web-enabled computers. ABS was used initially in 2009 to supplement the RDD frame and subsequently replaced it. This change occurred in response to declining RDD response rates and the growing number of cell phone-only households particularly among young adults and minorities that are often excluded from the traditional, RDD, landline telephone frame. ABS allows probability-based sampling of addresses from the U.S. Postal Service s Delivery Sequence File. Individuals residing at randomly sampled addresses are invited to join KnowledgePanel through a series of mailings (in English and Spanish); non-responders are phoned when a telephone number can be matched to the sampled address. Household members who were randomly-selected can indicate their willingness to join the panel by returning a completed acceptance form in a postage-paid envelope, calling a toll-free hotline and speaking to a bilingual recruitment agent, or accessing a dedicated recruitment website. ABS, conducted throughout the year, is done without replacement. Addresses with matched telephone numbers from the former RDD recruitment samples (for the last five years of calling) are also removed to eliminate duplication. KnowledgePanel Latino SM KnowledgePanel Latino SM is another GfK innovation. This panel provides researchers with the most statistically representative panel of U.S. Latino households in existence, with about 9,500 Latino adults (of whom over 4,200 are Spanish-language participants). This panel includes unassimilated Latinos and Spanish-language dominant households as well as those without Internet access at home before joining the panel. GfK provides web-enabled computers and Internet service for the roughly 40% of Latinos without web access from home. KnowledgePanel LatinoSM enhances our panel that already includes English-speaking Latino households. The sample for KnowledgePanel Latino is selected using a dual-frame design. The main sample is recruited by mail using English and Spanish materials. The ABS sample is supplemented with a smaller RDD telephone recruitment that specifically targets high density Hispanic areas across the country to exclusively recruit additional Spanish-dominant households. Panel Survey Sampling For all new panel members, demographic information such as gender, age, race/ethnicity, income, education and, for Latino members, language proficiency are collected using an online profile survey. This information is used to determine eligibility for specific studies and eliminates the need for gathering basic demographic information in each panel survey. After this survey is completed, the panel member is regarded as active and ready to be sampled for other surveys. Additionally, all Hispanic panel members are asked a series of questions that can be used by clients to apply to an acculturation scale. Such questions include media use, country of birth, years in the U.S. and other attitude and values questions. Samples are drawn from among active members using a probability proportional to size (PPS) weighted sampling approach. Customized stratified random sampling based on profile data is also conducted, as required by specific studies. In other panels, it is not unusual that only panel members with certain characteristics are selected for a survey. This selectivity in sampling can skew the remaining panel membership demographics and affect 3

4 the representativeness of samples that are selected for later surveys. In September 2007, GfK received a patent (U.S. Patent No. 7,269,570) for its unique methodology for selecting multiple online survey samples from a panel to ensure representativeness for second and subsequent samples within the same sampling period (most participants are eligible for only one survey per week). This sampling methodology was developed by Knowledge Networks in recognition of the practical issue that different survey samples may target different panel subpopulations. The KnowledgePanel selection methodology, which has been used by GfK since 2000, corrects for in-panel sampling and assures that multiple sequential KnowledgePanel samples from a finite panel membership can each reliably represent the U.S. population. Survey Frequency & Burden To minimize panel attrition, surveys are usually kept short (from 5 to 20 minutes in length). For surveys requiring 16 or more survey minutes, survey participation is rewarded with a variety of incentives (depending on length and difficulty, we most commonly offer points that respondents can accumulate that can be redeemed for cash awards, gift prizes, or sweepstakes opportunities). Further, steps are taken to ensure that panel members are not overburdened with many survey requests. The primary sampling rule is to assign only one survey per week to members. This level of survey frequency helps keep panelists engaged. On average, most GfK panelists participate in about two surveys a month. Some panelists may be in higher demand, depending on the projects being fielded. GfK operates a Panel Relations program to encourage participation and continuing member loyalty independent of survey completion. In this program, members can enter special raffles or be entered into special sweepstakes with both cash and other prizes to be awarded. Response Rates As a member of the American Association of Public Opinion Researchers (AAPOR), GfK follows the AAPOR standards for response rate reporting. However, the AAPOR standards were established for single survey events and not for Web or other longitudinal panels. Callegaro and DiSogra (2008) developed examples of response rates calculated for KnowledgePanel surveys and detailed a number of response formulae. In addition, they developed the concept of completion rate applicable to Web panels. The completion rate for KnowledgePanel is 65% with some variation depending on survey length, topic, and other fielding characteristics. In contrast, non-probability, opt-in, online panels typically achieve a survey completion rate in the 2% to 16% range. As a result of these standard metrics, panels can be more adequately compared (total number of panel members alone is an incomplete measure of a panel s scalability ). The effective panel size also depends on this completion rate. For example, GfK s KnowledgePanel requires only about 1,850 panelists to obtain 1,200 completed surveys, whereas a non-probability, opt-in sample may require a starting sample of 50,000 invitations or more to obtain 1,200 completed surveys. Thus, KnowledgePanel s size of approximately 55,000 can be comparable to an effective opt-in panel size of 1.7 million (assuming a 2% completion response rate for the opt-in panel). Statistical Weighting To correct for any biases in sampling or non-response, we compute sample weights in several stages. The design for KnowledgePanel recruitment begins as an equal probability sample with several enhancements incorporated to improve efficiency. Since any alteration in the selection process is a 4

5 deviation from a pure equal probability sample design, statistical weighting adjustments are made to the data to offset known selection deviations. These adjustments are incorporated in the sample s base weight. There are also several sources of survey error that are an inherent part of any survey process, such as non-coverage and non-response due to panel recruitment methods and to inevitable panel attrition. We address these sources of sampling and non-sampling error by using a panel demographic poststratification weight as an additional adjustment based on demographic distributions from the most recent data from the Current Population Survey (CPS). This weighting adjustment is applied prior to the selection of any client sample from KnowledgePanel, and these weights are used in the stratified, weighted, selection procedure for drawing samples from the panel. All the above weighting is done before the study sample is drawn. Once a study sample is finalized (all data collected and a final data set made), a set of study-specific post-stratification weights are constructed so that the study data can be adjusted for the study s sample design and for survey nonresponse. Starting with each panel member s base weight, an iterative raking procedure is used to achieve an optimal approximation of the relevant benchmarks to make survey respondents representative. References Callegaro, Mario & Charles DiSogra Computing Response Metrics for Online Panels. Public Opinion Quarterly 72: Yeager, David S., Jon A Krosnick, LinChiat Chang, Harold S Javitz, Matthew S Levendusky, Alberto Simpser, Rui Wang Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-probability Samples. Public Opinion Quarterly 75: Other methodological papers related to KnowledgePanel are available at 5