I Got a Feeling: Comparison of Feeling Thermometers with Verbally Labeled Scales in Attitude Measurement Randall K. Thomas, GfK Government and Academic Research & John Bremer, Toluna USA Presented at the 67th Annual Conference of AAPOR Orlando, FL May 16 May 20, 2012 1
Introduction Attitudes have a number of components, including affective, intention, and behavior. When assessing affective aspects of attitude, we a number of choices for response formats. There has been an ongoing debate in measurement over the optimum number of response options that can adequately capture the variance of people s responses. Some research has indicated that increasing number of response categories will improve both measurement reliability and validity (e.g. Alwin, 1997). Other research has indicated that the optimal number of categories may be between 5 to 11 categories, depending on topic and purpose of measurement. 2 2
Introduction In an attempt to develop measures that had ratio properties (with a natural zero point), the development of the Visual Analog Scale converted measurement into a 0 to 100 or 0 to 10 scale. Ratio scales were seen as having more powerful statistical advantages than the scales typically used in social science that are more at the interval or ordinal levels. Following the development of the VAS, the feeling thermometer was developed to assess affective aspects of attitude. This measure asked individuals to locate their affective experience (liking-disliking) on a scale ranging from warm (like) to cold (dislike) that had an underlying metric ranging from 0 to 100 (or in some cases 0 to 10). 3 3
Introduction Feeling thermometers were first incorporated into measurement of political figures and issues in the 1960s in the American National Election Survey. For all the presumed advantages of feeling thermometers, some have found that respondent use of them didn t not fully utilize the advantages of a ratio scale (Brady, 1985), and respondents used the measure in idiosyncratic ways. Our interest in this study was to compare the relative effectiveness of feeling thermometers with single response formats within web surveys, including an examination of a number of measurement characteristics. 4 4
Study 1 Method Respondents: Respondents were from the Harris Interactive non-probability panel and were randomly selected within age and sex strata to resemble U.S. population proportions. Waves of Survey Administration: 3839 respondents in July, 2000 wave 5606 respondents in October, 2000 wave Recontact Wave: We recontacted all participants in the pre-election waves immediately following the election in November, 2000 to participate in a follow-up survey. 1687 from Wave 1 and 2842 from Wave 2 completed the follow-up survey. 5 5
Study 1 Method Compared ratings of 3 political figures in 2000: George W. Bush Al Gore Bill Clinton Respondents were randomly assigned to use a feeling thermometer ranging from 0 to 100 or a 7 bipolar response format. Due to experimental requirements, 2/3 of respondents were assigned the feeling thermometer. 6 6
Study 1 Method Single Response Please rate how much you like each of the people listed below, using the following scale. 1 = Very much dislike 2 = Dislike 3 = Somewhat dislike 4 = Neither like nor dislike 5 = Somewhat like 6 = Like 7 = Very much like This scale required the respondent to enter a number in a numeric box next to each political figure. 7 7
Study 1 Method Feeling Thermometer Please rate how favorable or unfavorable you feel toward each person by picking a number between 0 and 100. The larger the number you pick, the more you like the person. Ratings between 50 and 100 mean that you feel favorable toward the person, and ratings between 0 and 50 mean that you feel unfavorable toward the person. You would rate a person at 50 if you don t feel favorable or unfavorable. Please enter the number next to each person s name. This scale required the respondent to enter a number in a numeric box next to each political figure. 8 8
Study 1 Results 9 9
July 2000 Wave Results Means 10 10
October 2000 Wave Results Means 11 11
Results Distribution Overall Endorsement Proportions Feeling Thermometer 12 12
Results Distribution Overall Endorsement Proportions Single Response 13 13
Results Test-retest Test-retest reliability was approximately the same for the feeling thermometer as for the single response format. Response Format Politician July October Bill Clinton 0.883 0.880 101 Feeling Thermometer Al Gore 0.840 0.859 George W. Bush 0.843 0.863 Bill Clinton 0.871 0.881 7 Single Response Al Gore 0.805 0.853 George W. Bush 0.812 0.872 These correlations are between pre-election and post-election evalutions. 14 14
Results Validity Party ID We used political party identification as a primary criterion. Party ID is measured as a branching question asking for party ID first and, if identify as Democrat or Republican ask strength, if identify as Other or Independent ask direction of lean. The responses can then be scored as a 1 to 7 variable. We regressed each political figure rating on party ID by response format. 15 15
Results Criteria-related Validity July 2000 - Political Party ID Criterion 16 16
Results Criteria-related Validity October 2000 - Political Party ID Criterion 17 17
Results Criteria-related Validity Bush Vote Criterion: Based on self-reports in the post-election wave, we next created a criterion variable to establish a vote strength measure to use as a primary dependent variable for determining relatedness of the evaluative responses to behavior: 1 = Voted for George W. Bush.5 = Did not vote, but would have voted for Bush 0 = Did not vote for Bush or if voted would not have voted for Bush 18 18
Results Criteria-related Validity July 2000 Vote Criterion r 2 of evaluation predicting Bush vote choice 19 19
Results Criteria-related Validity July 2000 Vote Criterion r 2 of evaluation predicting Bush vote choice 20 20
Study 2 Method Fielded February to October, 2008 on a monthly basis. 17,396 respondents from Harris Interactive panel randomly selected from strata to resemble U.S. population in terms of age, sex, education, income, region, and race/ethnicity Politicians to evaluate Hillary Clinton Barack Obama John McCain George W. Bush Bill Clinton 21 21
Study 2 Method 2 separate studies: a. February to July, 2008 4 response formats b. August October, 2008 5 response formats 22 22
Study 2 Method Single response formats: 1. Unipolar Single Response o Do not like at all o Like somewhat o Like o Like strongly o Like very strongly o Not at all familiar 2. Bipolar Single Response o Dislike strongly o Dislike somewhat o Neither like nor dislike o Like somewhat o Like strongly o Not at all familiar Single response formats were presented as Grids with the political figures in rows and the responses in columns. 23 23
Method Thermometer Formats: 3. Bipolar Thermometer Please use a scale of 0 to 100 where 0 means Dislike strongly, 50 means Neither like nor dislike, and 100 means Like strongly. 4. Unipolar Thermometer Neutral Midpoint Please use a scale of 0 to 100 where 0 means Do not like at all, 50 means Neither like nor dislike, and 100 means Like very strongly. 24 24
Study 2 Method Thermometer format (Study 2-b) 5. Unipolar Thermometer Like Midpoint Please use a scale of 0 to 100 where 0 means Do not like at all, 50 means Like moderately, and 100 means Like very strongly All Feeling Thermometers required a number to be entered into a numeric box, or select a Not Familiar box to the right of the numeric boxes. 25 25
Results Study 2 26 26
Results Time to Complete Studies 2a and 2b The Thermometer scales took significantly longer than the single response grids. No significant differences in time to complete for the thermometer scales. 27 27
Results Time to Complete Time to Complete by Response Format in seconds 28 28
Results Time to Complete Time to Complete by Response Format in seconds 29 29
Studies 2a and 2b Results - Means Bipolar single response means were found to be significantly higher than all thermometers, except for George W. Bush. Unipolar single response means were significantly lower than all other measures for all targets. We did not find any significant differences in the means for thermometers relative insensitivity for response labels. 30 30
Study 2a Results - Means 31 31
Study 2b Results - Means 32 32
Results - Variance Study 2a Standard Deviations 33 33
Results - Variance Study 2b Standard Deviations 34 34
Results Validity Party ID Political Figures Other than Major Party Candidates: Regressing evaluations on the 7 category Party ID, we found significantly higher r 2 for George W. Bush for all measures, and bipolar single response significantly higher overall for Study 2a, but not Study 2b. 35 35
Results Validity Party ID Study 2a February July 2008 36 36
Results Validity Party ID Study 2b August October 2008 37 37
Results Validity Party ID Political Figures for Major Party Candidates: Regressing evaluations on the 7 category Party ID, we found lower values of r 2 for all measures, with evaluations of John McCain less predictive of party ID than Barack Obama in Study 2a. In Study 2b, this relationship was obtained again, but all r 2 values were significantly higher closer to the election (unlike those found for the non-candidates). 38 38
Results Validity Party ID Study 2a February July 2008 r 2 evaluations predicting Party ID 39 39
Results Validity Party ID Study 2b August October 2008 r 2 evaluations predicting Party ID 40 40
Results Validity Vote Choice Created a vote choice variable to serve as a primary criterion using John McCain as target: 1.00 Likely to vote, chooses McCain on first.75 Likely to vote, chooses McCain on lean.50 Not likely to vote, chooses McCain on first.25 Not likely to vote, chooses McCain on lean 0.00 Did not choose McCain Regressed ratings of political figures on vote choice by response format and wave. 41 41
Results Validity Vote Choice For the political figures who were not major party candidates, evaluations of George W. Bush was more predictive of vote choice, and evaluations of the political figures were more predictive of vote choice when the measures were assessed closer to the election (Study 2b). 42 42
Results Validity Vote Choice Study 2a February July 2008 r 2 evaluations predicting Vote Choice 43 43
Results Validity Vote Choice Study 2b August October 2008 r 2 evaluations predicting Vote Choice 44 44
Results Validity Vote Choice For the political figures who were major party candidates, evaluations of the political figures were more predictive of vote choice the closer that evaluations were made to the election (Study 2b). 45 45
Results Validity Vote Choice Study 2a February July 2008 r 2 evaluations predicting Vote Choice 46 46
Results Validity Vote Choice Study 2b August October 2008 r 2 evaluations predicting Vote Choice 47 47
Conclusion Feeling thermometers take much longer for respondents to complete. General insensitivity to labels used on feeling thermometers (end labels, midpoints) was found. Feeling thermometers show no clear advantage over single response formats in means, variance, reliability, or validity. Increased number of responses available does not necessarily improve measurement accuracy. At least for web-based surveys, the use of feeling thermometers may not be warranted based on respondent burden alone. 48 48
Thank You! Randall K. Thomas randall.thomas@gfk.com 49 49