A comparison of reliability between telephone and web-based surveys

Transcription

1 A comprison of relibility between telephone nd web-bsed surveys Contct Author: Krin Brunsberger, Ph.D., Associte Professor of Mrketing University of South Florid St. Petersburg College of Business Administrtion 140 Seventh Avenue South, Dvis Hll 258 St. Petersburg, FL Telephone: (727) Fx: (727) E-mil: Hns Wybeng 151 Frmington Avenue MB62 Hrtford, CT Telephone: (860) E-mil: Roger Gtes DSS Reserch 6750 Locke Avenue Ft. Worth, TX Telephone: (817) E-mil:

2 Abstrct The purpose of the present study ws to compre methodologiclly sound telephone interviewing technique to the most promising web survey technique. Specificlly, the web survey conducted for the present study rndomly selected respondents who were members of consumer pnel wheres the telephone survey used cold-clling method to rndomly selected respondents. Two wves of ech telephone surveys nd web pnel surveys were conducted. The present study is one of the first to empiriclly show tht web pnels cn produce more relible dt estimtes thn telephone surveys. Further, web pnels re cheper nd less time consuming to conduct thn telephone surveys. Even though web pnels might not be pproprite for ll survey reserch endevors, the results show tht they cn be vible lterntive to telephone surveys tht llows reserchers to conduct high-qulity reserch. Keywords: telephone surveys, web pnels; relibility of surveys.

3 1. Introduction A comprison of relibility between telephone nd web bsed surveys During the lst decde, reserchers utilizing telephone surveys to collect dt hve been fced with dilemm. On one hnd, the increse in telephone ownership in the United Sttes to current levels, with 97.6% of ll households hving telephones (SSI, 2003), hs mde telephone surveys very populr in mrketing reserch prctice s method of dt collection (Link nd Oldendick, 1999). On the other hnd, telephone surveys hve been found to hve limittions. Due to the tremendous increse in telemrketing nd telemrketing frud, certin groups of prospective telephone survey respondents (e.g., the most ffluent, households with older children) re employing gtekeeper technologies (i.e., Cller-ID nd nswering mchines) to screen nd eliminte cllers (Tuckel nd O Neill, 2002), nd those who nswer their phones re quick to hng up if they suspect cller might be trying to sell something (Elvin, 2000; Remington, 1992). As result, telephone interviewers often do not even get chnce to explin tht they re conducting survey (Council for Mrketing nd Opinion Reserch, 2003), nd response rtes hve stedily declined (Keeter, Miller, Kohut, Groves nd Presser, 2000) to reported lows of 7% (Council for Mrketing nd Opinion Reserch, 2003). This decrese presents problem becuse not only does it increse the cost of conducting telephone surveys, but it lso leds to questions concerning the generlizbility of the results (Struebbe, Kernn nd Grogn, 1986; Tuckel nd O Neill, 2002). In ddition, telephone surveys, due to perceived lck of nonymity, hve been found to led to decresed completeness of reporting (Cooley, Miller, Gribble nd Turner, 2000) nd higher levels of respondent editing thn certin other types of survey modes (Johnson, Fendrich nd Shligrm, 2000). Further, the immedite ttention these surveys require hs been suggested to mke potentil respondents reluctnt to prticipte (Melymuk, 1997). In recent yers, the focus hs shifted to web surveys which cn overcome some of the limittions of telephone surveys, re lower in cquisition cost nd time (Krsilovsky, 1996), nd llow ccess to high numbers of potentil respondents (Couper, 2000). Online reserch hs been estimted to ccount for pproximtely 50% of ll mrketing reserch revenues by the yer 2005 (Jmes, 2000). Web surveys, however, hve been suggested to be fr from perfect (Gormn, 2000). Tht is, nonresponse rtes cn be just s high when using web surveys s they re with telephone surveys, coverge error (i.e., function of the difference between the defined trget popultion of interest nd the popultion frme obtined used to study this trget) my be high (Couper, 2000), nd respondents my flsify their demogrphic informtion nd thus undermine the ccurcy of the results nd the vlidity of conclusions drwn from the reserch. The use of pnels specificlly recruited for online reserch, though, cn mitigte these weknesses of web surveys (Jmes, 2000). To summrize, prior reserch shows tht both telephone nd web surveys offer number of dvntges nd lso suffer from number of limittions. Little reserch, however, formlly compres telephone nd web surveys in terms of resultnt dt relibility. The uniqueness nd criticl importnce of the present study is tht it is one of the first ttempts to empiriclly investigte this issue by compring methodologiclly sound nd commonly used telephone interviewing technique to the most promising web survey technique. Tht is, the web survey conducted for the present study rndomly selected respondents who were members of consumer pnel (i.e., web pnel) wheres the telephone survey used cold-clling method to rndomly selected respondents. Further the present study is unique in tht it is, to the best of our knowledge, the only one so fr tht utilizes two wves of dt collection for both telephone 1

4 surveys nd web pnels, nd thus llows to exmine test-retest relibility (McDniel nd Gtes, 2005). 2. Literture review 2.1. Reserch compring survey methods Much reserch compring different survey methods hs been published. Due to the reltive newness of web surveys, much of this reserch hs focused on compring personl interviews to mil surveys (e.g., Abdel-Aty, 2003), telephone to mil surveys (e.g., Wlker nd Restucci, 1984), nd mil to e-mil surveys (e.g., McDonld nd Adm, 2003). So fr, little reserch hs compred web-bsed surveys to telephone surveys. To the best of our knowledge, the only exception to dte ws published in Europe by the Interntionl Journl of Mrket Reserch. In terms of smple chrcteristics, the uthors Roster, Rogers, Album, nd Klein (2004) find their U.S. telephone respondents, in comprison to their U.S. web pnel respondents, to be older, more likely to be Hispnic, nd to be better educted. With regrd to cost, comprison shows tht ech telephone survey cost, on verge, $30, wheres the price tg for ech web-bsed survey ws, on verge, $ In terms of response effects, the uthors report tht web pnel respondents, s compred to telephone survey respondents, hve tendency to give more neutrl or negtive ttitudinl evlutions, to use more simplistic cognitive response style, nd for their responses to be slightly more relible. The uthors further report tht the web pnels, in comprison to the telephone surveys, result in more nonresponses to some of the questions. Thus, the design pprently filed to incorporte one of the benefits of web surveys, nmely the possibility to not permit respondents to skip items (see Couper, Trugott, nd Lmis, 2003). Considering tht web surveys where skips re not permissible feture built-in check tht cn reduce respondent errors (Dillmn, 2000; McCullough, 1998), this ppers to be design flw which points to the need for more reserch in this re Representtiveness of respondents Even though 97.6% of ll households hve telephones (SSI, 2003), the bove discussion shows tht telephone survey respondents might not be representtive of mrketer s popultion of interest, nd telephone survey results might thus suffer from lck of generlizbility. Internet reserch hs been suggested to suffer from the sme problems due to coverge error (Couper, 2000). A closer look t this issue, however, shows tht ccess to the Internet hs been rpidly incresing nd is vilble to nywhere from 67.9% to 75% of the U.S. popultion (Fdner nd Mndese, 2004). Further, some of the demogrphic differences in terms of ccess to the Internet tht were observed during the 1990s (i.e., gender, rurl/urbn popultion) hve vnished by now (Zhng, 2000). Since certin groups re still underrepresented including lower income groups, the less educted, Africn Americns nd Hispnics (Chudhuri, Flmm nd Horrign, 2005) online reserchers utilizing pnels hve to mke sure tht pnel members re representtive of the defined trget popultion of interest. In other words, even though some suggest tht with telephone-bsed surveys it is possible to obtin representtive ntionl smple, wheres internet surveys re by definition restricted to respondents with internet ccess (Roster et l., 2004, p. 360), the evidence shows tht reserchers using either need to ensure representtiveness of their smples Use of interviewers 2

5 The mjor dvntge of using interviewers to conduct surveys is tht converstionl interviewing techniques cn mke sure respondents understnd the questions, thus ensuring uniform interprettion of the intent of ech question which should led to more ccurte responses (Conrd nd Schober, 2000). On the downside, this technique cn led to bised results (Betty, 1995) unless ech interviewer hndles nd interprets ech question in the exct sme mnner. This interviewer bis or effect, which hs been defined s the tendency for nswers provided by the respondent nd recorded in questionnire to vry depending on which interviewer is ssigned to the respondent (Johnson, Fendrich nd Shligrm, 2000, p. 77), hs been detected in telephone surveys (Johnson, Fendrich nd Shligrm, 2000; Lueptow, Moser nd Pendleton, 1990). Evidence shows tht interviewer gender nd rce re likely to hve n impct in telephone surveys (Johnson, Fendrich nd Shligrm, 2000; Lueptow, Moser nd Pendleton, 1990), s does the technique used nd the behvior displyed by the telephone interviewer (Betty, 1995; Keeter et l., 2000). Another problem relted to interviewers is termed misbehvior by interviewers (Kiecker nd Nelson, 1996) which refers to ctivities tht re dishonest, the product of low morle, interviewer lck of understnding of the importnce of the interviewing process, nd/or n interviewer s ttempt to improve reserch qulity. Kiecker nd Nelson (1996) report the outcome of self-report study where telephone interviewers employed by U.S. firms listed s suppliers of Interviewing Services were questioned in terms of their interviewing behvior. The results show tht both fbriction of item response nd entire interviews re reltively rre (.8% nd.4%, respectively). However, Kiecker nd Nelson s (1996) subsequent field experiment tht mde use of plnted respondents who were interviewed by 33 reserch firms unwre of the purpose of the experiment shows more worrisome picture. The ctul incidence of telephone interviewer misbehvior is much higher thn indicted by self-report dt. Actul incidence of fbriction item response ws 23.6% nd the incidence of entire interview fbriction ws 4.9%. 3. Hypothesis 3.1. Dt relibility As indicted by the bove discussion, the elimintion of interviewers in web-bsed surveys is likely to led to some very significnt dvntges. Tht is, becuse web surveys re self-dministered nd the dt entered by respondents re generlly utomticlly fed into spredsheet without ny further humn intervention (McDniel nd Gtes, 2005), they re free of interviewer bis nd misbehvior, provide privcy to the respondent nd thus re likely to led s compred to telephone surveys to incresed completeness of reporting of some types of behviors nd re further likely to reduce certin types of response errors, such s socil desirbility nd prestige (Cooley et l., 2000; Hir, Bush nd Ortinu, 2003; Johnson, Fendrich nd Shligrm, 2000). Web surveys re thus likely to produce more stble mesures thn telephone surveys, s shown by Roster et l. (2004) who report tht using the coefficient thet to ssess the relibility of their fctor nlyses shows tht web survey respondents produce dt tht re slightly more relible (thet =.92) thn those generted by telephone survey respondents (thet =.84) (see Crmines nd Zeller, 1979). In ddition, web-bsed surveys re expected in comprison to telephone surveys to produce more consistent results over time. Consistency over time is n indictor of relibility (McDniel nd Gtes, 2005), nd thus web-bsed surveys re expected to generte more relible results thn telephone surveys due to the lck of interviewers. This expecttion will be tested in 3

6 the following hypothesis. Considering tht the current study is compring telephone surveys to very specific form of web-bsed surveys nmely web-bsed pnel surveys the terminology from this point on will reflect this peculirity: 4. Methods H1: Web-bsed pnel surveys show significntly higher level of dt relibility thn telephone surveys. As mentioned erlier, the purpose of the present study is to compre commonly used nd methodologiclly sound telephone interviewing technique (i.e., rndom digit diling) to promising web survey technique (i.e., web pnel). The focus of the comprison is on the dt relibility of these two specific interviewing methods Dt collection To obtin the necessry dt for testing the study s hypothesis, two-wve dt collection frmework ws incorported using both the telephone nd Internet s communiction medi. The initil wve of reserch for both survey methods ws conducted in June-July 2002, nd the second wve for both survey methods ws completed in erly September In efforts of mintining control in the dt collection tsks, the sme commercil reserch house ws retined to conduct both web-pnel nd telephone surveys for both time periods. In ddition, ll dt were collected in the northest region of the United Sttes. The questionnire consisted of totl of 21 questions designed to ssess current nd previous helth insurnce coverge, how desirble certin ttributes of insurnce provider re, nd demogrphic chrcteristics. The content of both web-pnel nd telephone surveys ws identicl. Telephone dt collection: A rndom smpling procedure ws used to drw respondents for both wves of telephone interviewing. A screening question ws used to mke sure only qulified respondents prticipted nswered the survey (i.e., dults who hve privte group helth insurnce through n employer). All interviews were conducted from centrl loction, nd for control purposes the interviewing process ws unobtrusively monitored by supervisory personnel. Approximtely 30 interviewers were used for this project. All were professionl telephone interviewers who were well-trined nd work on regulr bsis for the reserch compny tht conducted the interviewing. The smpling list ws purchsed from mjor provider of such lists. Contct procedures included four contcts (i.e., the initil contct nd three cllbcks) to individul respondents, if necessry. Usble responses were 802 for Wve 1, nd 801 for Wve 2 (see Tble 1 for informtion on response rtes). Web pnel dt collection: The web pnel list ws obtined from the sme supplier of the telephone smpling lists; however, there ws no overlp between the smpling lists for telephone nd web pnels. All members of the web pnel hd previously greed to prticipte in reserch projects. The pnel list included totl of 1.5 million households tht were demogrphiclly mtched to the defined trget popultion of the geogrphic re in question. The respondents for Wve 1 nd Wve 2 were rndomly drwn from the pnel. The protocol for pnel members in the trgeted re included four contcts (i.e., the initil invittion, nd three reminders). Usble 4

7 responses were 853 for Wve 1, nd 929 for Wve 2 (see Tble 1 for informtion on response rtes). 5. Dt nlyses nd results 5.1. Demogrphic profiles of the smples [INSERT TABLE 1 HERE] Demogrphic profiles of ll smples cn be found in Tble 2. Tble 2 shows some demogrphic differences between the web pnel nd telephone survey respondents, which is not n unusul observtion (see Roster et l., 2004). To further compre the rndom telephone nd web-pnel smples, respondents were sked bout the type of their current privte helth insurnce coverge. The results for both the web pnels nd the telephone surveys in both dt collection wves cn be found in Tble 3. The dt provided in the secondry dt column of Tble 3 were developed from informtion vilble through InterStudy (2003). Comprison of this informtion with the telephone survey nd web pnel results shows tht the web pnel results re more consistent with the results from secondry dt in tht helth insurnce coverge results for the web pnels more closely conform to the estimtes developed from secondry dt thn the results from the telephone surveys. Further, the web pnel results lso pper to be more congruent between wves 1 nd 2 thn the telephone survey results Hypotheses testing [INSERT TABLES 2 AND 3 HERE] The hypothesis postultes tht web-bsed pnel surveys led to significntly more relible results thn telephone surveys. To test this hypothesis, respondents were sked to rte crriers of helth insurnce on ten ttributes in terms of the desirbility of ech ttribute (see Tble 4; five-point scles from 1 = not desirble to 5 = extremely desirble). One-wy ANOVA results show sttisticlly significnt differences on ll ten ttributes when compring the rtings of the telephone survey wves 1 nd 2 (see Tble 5), wheres sttisticlly significnt difference shows on only one of the ten ttributes cross web pnel wves 1 nd 2 (see Tble 6). Thus, the hypothesis cn be confirmed becuse web pnel results re more congruent cross wves, nd thus more relible, thn telephone survey results. [INSERT TABLES 4, 5, nd 6 HERE] 6. Discussion, implictions, nd suggestions for future reserch 6.1. Compring dt relibility A mjor contribution to the literture provided by this study is tht it provides empiricl evidence tht shows tht web pnels disply higher levels of dt relibility thn telephone surveys. As hypothesized, this effect is likely to be due to the lck of interviewers in web pnel dministrtion, which cncels out the interviewer effect or bis nd ffords more privcy to the respondents. Prior reserch suggests tht mny individuls consider informtion on helth issues 5

8 to be of privte nture nd re more likely to serch the Internet for such informtion rther thn sk their helthcre providers to furnish it (Fox nd Rinie, 2002). Similrly, mny of the respondents included in the current study might very well perceive informtion bout their helth insurnce, the topic of the present survey, to be eqully privte nd hesitnt to discuss such informtion with n interviewer. Hesitnt respondents feel justified to give inccurte informtion to n interviewer to protect their privcy (Tourngeu nd Smith, 1996), resulting in lower levels of dt relibility thn such informtion ssessed through self-dministered survey. A criticl need for more empiricl reserch on investigting the types of informtion consumers re hesitnt (versus comfortble) to disclose to interviewers cn be identified. More reserch is lso needed to gin clerer understnding nd insight to the extent to which telephone survey respondents unduly rely on their memories when sked to provide informtion, wheres, web pnel respondents hve the luxury of checking relevnt informtion before nswering specific survey items. In ddition, more empiricl reserch is needed to investigte whether knowledge questions plce n undue burden on telephone respondents, unless of course, the survey s focus is to ssess top-of-the-mind knowledge. Further, future reserch should investigte the impct of survey pproch (web versus telephone) versus smpling procedures (pnel versus cold-clling) on response rtes nd dt ccurcy (see Jmes, 2000). Here, it is lso of interest to investigte to wht extent respondents to web-pnel surveys, when compred to those of telephone surveys, experience n incresed sense of privcy Compring the cost nd time of dt collection The figures for clculting overll dt cquisition costs do not include the costs for designing the survey, progrmming the survey, or nlyzing the dt, becuse these costs re the sme for both pproches. Since the dt for both wves of telephone nd web-pnel surveys were collected by the sme commercil reserch house, the overll dt cquisition costs for telephone surveys re bsed on interviewer wges, plus n lloction for overhed nd profit. For the web pnels, overll dt cquisition costs include the cost of contcting pnel members nd directing them to the survey site, plus n lloction for overhed nd profit. Accordingly, the cost of ech telephone survey totled $22.75, wheres the cost of conducting ech web-pnel survey totled $6.50. Tht is, in terms of overll dt cquisition costs, the cost of ech web-pnel survey mounted to only 29% of the cost of ech telephone survey. With regrd to the telephone surveys, the figures for clculting overll dt cquisition time re bsed on the use of 30 CATI sttions nd contct procedure tht included up to three cllbcks to individul respondents, if necessry. For the web pnels, overll dt cquisition time is bsed on the ctul time it took to send invittions to the web pnelists, nd the time it took pnelists to visit the survey site nd complete the survey. Accordingly, ech web-pnel survey ws completed in n verge of three dys while ech telephone survey took nerly two weeks to complete, on verge. In other words, in the present study, ech web-pnel survey ws completed, on verge, in bout 21% of the time it took to complete ech telephone survey, on verge Differences in ttitude? A closer look t Tble 4 shows tht web pnel respondents, in comprison to telephone survey respondents, hve tendency to be more demnding in terms of the requirements they plce on their helth insurnce provider. These differences re sttisticlly significnt on 8 out of 10 ttribute rtings. This is in line with Roster et l. s findings (2004) tht web pnel 6

9 respondents, when compred to telephone survey respondents, tend to disply more neutrl or negtive ttitudinl evlutions. Future reserch is needed to investigte whether these findings cn be replicted. Future reserch lso needs to ddress tht the nswers given by telephone survey respondents disply higher levels of vribility (see Tble 4) thn those given by web pnel respondents. This is n issue of dt vlidity nd reserch needs to investigte which survey pproch is more vlid under wht circumstnces. 7

10 References Abdel-Aty MA. Hybrid distribution nd response techniques for n origin-destintion trvel survey. ITE Journl 2003;73(2): Betty P. Understnding the stndrdized/non-stndrdized interviewing controversy. J Officil Sttistics 1995;11: Chudhuri A, Flmm KS, Horrign J. An nlysis of the determinnts of internet ccess. Telecommunictions Policy 2005;29: Conrd FG, Schober MF. Clrifying question mening in household telephone survey. Pub Opinion Q 2000;64(1):1-28. Cooley PC, Miller HG, Gribble JN, Turner CF. Automting telephone surveys: using T-ACASI to obtin dt on sensitive topics. Compters Hmn Behvior 2000;16:1-11. Council for Mrketing nd Opinion Reserch respondent coopertion nd industry imge survey. Cincinnti, OH: CMOR; Couper MP. Web surveys: review of issues nd pproches, Pub Opinion Q 2000;64: Dillmn DA. Mil nd internet surveys: the tilored design method. New York (NY): John Wiley nd Sons Elvin J. Hi, my nme is (click!). Insght on the News 2000;16(10)(Mrch 13):34. Fdner R, Mndese J. Net-rtings: Internet is 75% full; comscore; they re full of it. MediPost s Medi Dily News ( newsdte=03/19/2004). Fox S, Rinie L. E-ptients nd the online helth cre revolution. Physicin Executive 2002;28(6): Gjestlnd L. Net? not yet. Mkting Resrch 1996;8(1): Hir JF Jr, Bush RP, Ortinu DJ. Mrketing reserch: within chnging informtion environment. New York (NY): McGrw-Hill; InterStudy. The InterStudy competitive edge St. Pul (MN): InterStudy Publictions. Jmes D. The future of online reserch. Mrktng Nws 2003(Jnury 3):1. Johnson TP, Fendrich M, Shligrm C. An evlution of the effects of interviewer chrcteristics in n RDD telephone survey of drug use. J Drug Issues 2000;30(1): Keeter S, Miller C, Kohut A, Groves RM, Presser S. Consequences of reducing nonresponse in ntionl telephone survey. Pub Opinion Q 2000;64(2): Kiecker P, Nelson JE. Do interviewers follow telephone survey instructions? J Mrk Res Soc 1996;38(2): Krsilovsky P. Surveys in cyberspce. Amricn Demogrpics 1996;(Nov Dec): Link MW, Oldendick, RM. Cll screening: is it relly problem for survey reserch? Pub Opinion Q 1999;63(4): Lueptow LB, Moser SL, Pendleton BF. Gender nd response effects in telephone interviews bout gender chrcteristics. Sex Roles 1990;22(1-2): McCullough D. Web-bsed mrket reserch ushers in new ge. Mktg Nws 1998;32(19): McDniel C, Gtes R. Mrketing Reserch. Hoboken, NJ: Wiley; McDonld H, Adm S. A comprison of online nd postl dt collection methods in mrketing reserch. Mktng Intllgnce nd Plng 2003;21(2): Melymuk K. Not nother #$!&!$ survey! Computerworld 1997;31(November 24):82. Remington T. Telemrketing nd declining survey response rtes. J Adv Res 1992;32(3):RC6-7. SSI. Sttistics supplied by SSI on 3/10/

11 Roster CA, Rogers RD, Album G, Klein D. A comprison of response chrcteristics from web nd telephone surveys. Interntl J of Mkt Rsch 2004;46(3): Struebbe JM, Kernn JB, Grogn TJ. The refusl problem in telephone surveys. J Adv Res 1986;June/July: Tourngeu R, Smith TW. Asking sensitive questions: the impct of dt collection mode, question formt, nd question context. Pub Opinion Q 1996;60: Tuckel P, O Neill H. The vnishing respondent in telephone surveys. J Adv Res 2002;42(5): Wlker AH, Restucci JD. Obtining informtion on ptient stisfction with hospitl cre: mil versus telephone. Helth Svcs Rsrch 1984; 19(3): Zhng Y. Using the internet for survey reserch: cse study. J Americn Society for Informtion Science 2000;51(1):

12 Tble 1 Detils bout Response Rtes Telephone Wve 1 Wve 2 N Percentge N Percentge Attempted to rech 2, , Never reched Reched 2, , Initil refusl 562 percentge of those ttempted to rech percentge of those ttempted to rech percentge of those reched percentge of those reched Qulified for survey 1,129 percentge of those ttempted to rech ,272 percentge of those ttempted to rech percentge of those reched percentge of those reched percentge of those reched who did not initilly refuse percentge of those reched who did not initilly refuse Refused fter qulified 327 percentge of those ttempted to rech percentge of those ttempted to rech percentge of those reched percentge of those reched percentge of those reched who did qulified percentge of those reched who did qulified Completed survey (usble) 802 percentge of those ttempted to rech percentge of those ttempted to rech percentge of those reched percentge of those reched percentge of those reched who did qulified percentge of those reched who did qulified Web Pnel Invited to prticipte 2, , Clicked through survey 1, , Qulified for survey 1,038 Percentge of those invited to prticipte ,155 Percentge of those invited to prticipte Percentge of those who clicked through survey Percentge of those who clicked through survey Did not complete survey 185 Percentge of those invited to prticipte Percentge of those invited to prticipte 9.73 Percentge of those who clicked through survey Percentge of those who clicked through survey Percentge of those who qulified for survey Percentge of those who qulified for survey Completed survey (usble) 853 Percentge of those invited to prticipte Percentge of those invited to prticipte Percentge of those who clicked through survey Percentge of those who clicked through survey Percentge of those who qulified for survey Percentge of those who qulified for survey

13 Tble 2 Respondent Demogrphics Age Percentge Distribution Wve 1 Wve 2 Ctegories Phone Surveys Web Surveys Phone Surveys Web Surveys b or older c Mritl Sttus Percentge Distribution Wve 1 Wve 2 Ctegories Phone Surveys Web Surveys Phone Surveys Web Surveys Mrried/Living with Prtner Single Divorced Widowed b Number of Children under the Age of 18 Percentge Distribution Wve 1 Wve 2 Ctegories Phone Surveys Web Surveys Phone Surveys Web Surveys No children child c children children c children children Income Percentge Distribution Wve 1 Wve 2 Ctegories Phone Surveys Web Surveys Phone Surveys Web Surveys Less thn $35, $35,000 to $49, $50,000 to $74,999 b b $75,000 to $99, $100,000 or more significnt t.0001 level b significnt t.005 level c significnt t.05 level 11

14 Tble 3 Type of Current Helth Insurnce Coverge Percentge Distribution Telephone Surveys Web Surveys Secondry Dt Ctegories Wve 1 Wve 2 Wve 1 Wve 2 HMO PPO POS Trditionl/Indemnity/ Fee-for-Service Defined Contribution Pln

15 Tble 4 Desirbility Attribute Rtings Mens nd Stndrd Devitions Telephone Surveys Web Surveys Attribute Wve 1 Wve 2 Wve 1 Wve 2 Men Std Dev Men Std Dev Men Std Dev Men Std Dev Trustworthy Esy to work with Knowledgeble Well-respected by physicin Accessible nd responsive Relible Fst nd ccurte clims processing Good vlue for the money Access to the best physicins Quickly resolves problems/issues

16 Tble 5 ANOVA Results: Telephone Survey Desirbility Attribute Differences Across Wves 1 nd 2 Attributes Sum of Squres df Men Squre F Sig. Trustworthy Between Groups E-05 Within Groups Totl Esy to work with Between Groups E-06 Within Groups Totl Knowledgeble Between Groups E-05 Within Groups Totl Well-respected by physicins Between Groups Within Groups Totl Accessible nd responsive Between Groups E-10 Within Groups Totl Relible Between Groups E-06 Within Groups Totl Fst nd ccurte clims processing Between Groups E-09 Within Groups Totl Good vlue for the money Between Groups Within Groups Totl Access to the best physicins Between Groups Within Groups Totl Quickly resolves problems Between Groups E-06 Within Groups Totl

17 Tble 6 ANOVA Results: Web Survey Desirbility Attribute Differences Across Wves 1 nd 2 Attributes Sum of Squres df Men Squre F Sig. Trustworthy Between Groups Within Groups Totl Esy to work with Between Groups Within Groups Totl Knowledgeble Between Groups Within Groups Totl Well-respected by physicins Between Groups Within Groups Totl Accessible nd responsive Between Groups Within Groups Totl Relible Between Groups Within Groups Totl Fst nd ccurte clims processing Between Groups Within Groups Totl Good vlue for the money Between Groups Within Groups Totl Access to the best physicins Between Groups Within Groups Totl Quickly resolves problems Between Groups Within Groups Totl