Presidential Approval Ratings in Perspective

Transcription

1 Presidential Approval Ratings in Perspective Larry Hugick, Jonathan Best, and Stacy DiAngelo Princeton Survey Research Associates International Princeton Survey Research Associates International 911 Commons Way 1211 Connecticut Avenue NW Princeton, New Jersey Washington, D.C (609) (202) May 2004

2 Presidential Approval 2 PRESIDENTIAL APPROVAL RATINGS IN PERSPECTIVE... 1 ABSTRACT... 3 INTRODUCTION... 4 METHODOLOGY... 8 DATA... 8 COMPUTING QUARTERLY AND YEARLY AVERAGES... 9 STATISTICAL ANALYSES... 9 ANALYSIS AND RESULTS DIFFERENCES IN POLLING METHODOLOGY Interviewing Methods and Practices Sampling and Weighting Respondent Selection Questionnaire Issues Changes in Methodology during the Study Period TREND LINE FOR ALL POLLS PATTERNS OBSERVED IN YEARLY AVERAGES STATISTICAL ANALYSIS OF QUARTERLY AVERAGES ANOVA Procedures and Findings VISUAL REPRESENTATION OF POLL DIFFERENCES INVESTIGATING THE DON T KNOW FACTOR... 24\\ COMPARISONS WITH GALLUP IN-PERSON APPROVAL RATINGS CONCLUSIONS APPENDIX POLLING METHODS QUESTIONNAIRE PRESIDENTIAL APPROVAL BY ORGANIZATION: LINE CHARTS QUARTERLY AVERAGES BY ORGANIZATION ABC News/Washington Post CBS News CNN/USA Today/Gallup FOX News/Opinion Dynamics Newsweek... 47

3 Presidential Approval 3 Abstract This paper analyzes individual poll results, as well as quarterly and yearly averages for five major media polls the ABC News/Washington Post poll, the CBS News poll, the CNN/USA Today/Gallup poll, the FOX News/Opinion Dynamics poll, and the Newsweek poll in order to identify and explain differences in their presidential approval ratings during the period Differences in approval are relatively small. In a given year, the typical difference in percent approve across all polls is only four points. The level of don t know (DK), however, differs significantly by organization and analysis shows that reducing the DKs tends to increase approval without much affecting disapproval. The polls that use a conventional telephone survey methodology can be divided into two high DK polls (ABC and CNN) with a tendency toward higher approval scores and two low DK polls with a tendency toward lower scores (CBS and Newsweek). The FOX poll, which uses a somewhat different methodology, looks very close to the conventional high DK polls when its approval ratings are averaged over an entire year, but tends to register lower disapproval ratings, and its movement from quarter to quarter and poll to poll is less predictable than the others. Contemporary presidential job performance ratings from telephone polls seem to be reasonably close to those for past presidents taken from The Gallup Poll s historic inperson surveys. In the percent approve, however, today s high DK polls are slightly closer to the Gallup in-person results than today s low DK polls.

4 Presidential Approval 4 Introduction National polls have been asking the public to rate the president s job performance since 1935, during Franklin D. Roosevelt s first term in office. The most widely used version of this question Do you approve or disapprove of the way [president s name] is handling his job as president? was adopted by The Gallup Poll in While other question wordings are still employed, such as the one using a four-point scale (excellent, good, fair, or poor), the standard Gallup wording is now used in most, if not all, telephone polls sponsored by major national media organizations network broadcast and network TV news, as well as the largest national newspapers and news magazines. While 30 years ago political junkies hungry for the latest results might have to wait a week or two for a new round of presidential approval data, today hardly a day goes by without a new set of approval figures being served up by one organization or another. And that s not counting the steady stream of poll data provided by organizations with alternative lower-costs methodologies, like Rasmussen Reports and its automated polling that generates daily approval numbers. The availability of so many different measures of the public s evaluations of the president s job performance means that one poll with results that seem aberrant can be checked against others in a timely fashion. While more data is generally better than less data, it is not without its pitfalls. Most seasoned poll watchers know that the best practice is to limit trend analysis to polls of one particular organization rather than mix and match but there is a strong temptation to make cross-comparisons. Even if organizations use the same basic methodology and question wording, house effects caused by factors that are less

5 Presidential Approval 5 obvious such as the way a CATI system works and interviewers are instructed potentially introduce differences to poll results. Many political reporters, columnists and other non-survey research specialists who frequently use approval data know less than they should about differences in methodology between the major national media polls they monitor, and are misinformed about which differences are likely to have a significant impact on results and which are not. Variation in party ID distributions from poll to poll, the presumed political slant of the media organization sponsoring the poll, and purported day-of-the-week interviewing effects are examples of issues that are brought up all too frequently by those who know less than they think about poll methodology. Those of us who conduct these polls for the media often don t spend enough time keeping track of what our competitors are doing and our impressions of other pollsters methodology may not always reflect current practices. Through a careful study of recent presidential approval ratings by five of the most prominent national media polls, this paper seeks to provide information that will allow poll watchers to better interpret their results and put them in perspective. It attempts to answer the following questions: Are the approval ratings of these major media polls basically interchangeable, or do they differ in ways that can be quantified? If there are differences by organization, what factors are responsible? Is it reasonable to make a direct comparison between today s approval ratings based on telephone surveys and those for past presidents from Gallup s in-person surveys? More generally, the paper describes the current state of methodology of five major media polls and how closely their results correlate. 1 Edwards III, George C., with Alec M. Gallup. Presidential Approval: A Sourcebook (The Johns Hopkins University Press: 1990), page 3.

6 Presidential Approval 6 We used two criteria for determining which major media polls to select for this study: 1) the question posed had to make use of the standard Gallup approval question wording; and 2) approval figures had to be available from more than 12 individual polls sponsored by the media organization during the calendar year, excluding one-night polls i.e., with a frequency of more than once a month. The five polls meeting these two criteria are the ABC News/Washington Post poll; the CBS News poll; the CNN/USA Today/Gallup poll; the FOX News/Opinion Dynamics poll; and the Newsweek poll, conducted by Princeton Survey Research Associates International. 2 Of these five media polls, the CNN/USA Today/Gallup poll was by far the most active, releasing 41 separate sets of results over the 12-month period, while the ABC News/Washington Post poll was least active, with 16 approval measures in Two other prominent media polls that regularly ask the standard Gallup approval question the CNN/Time poll and NBC/Wall Street Journal poll fell below the activity frequency threshold. We were aware that the FOX News/Opinion Dynamics poll differed from the other four polls selected in their reporting base registered voters rather than the entire voting-age population. However, our own recent polling suggested that the registered voter base often representing over 80% of a general public adult sample introduces only small differences to poll measures of political attitudes. What s more, we believed the study would be enhanced by including a poll sponsored by top-rated FOX News cable network that has positioned itself as an alternative to the mainstream media, to see if their poll diverges in any specific way from the others. 2 In this report organization names are abbreviated as follows: ABC News/Washington Post ABC; CBS News CBS; CNN/USA Today/Gallup CNN; and FOX News/Opinion Dynamics FOX.

7 Presidential Approval 7 In defining a time period for this study, we sought to be as contemporary as possible to reflect current methods and the conditions that might affect our ability to conduct surveys, but at the same time providing sufficient data to analyze. In part, our decisions about which polls to study helped us make that decision, since the FOX News/Opinion Dynamics poll was not launched until Limiting the study to (the period encompassing Bill Clinton s second term in office and the first three years of George W. Bush s presidency) is a reasonable compromise, being reasonably upto-date but still providing a sizable data base of approval ratings for the five organizations. There is variety in that the two presidents being evaluated are from different political parties, one during his second term in office and the other during his first three years in office. We owe a debt of gratitude to our colleagues at other organizations whose polls were selected for inclusion in our study, whose cooperation in answering our questions and disclosing the details about their methods and practices added immeasurably to this research effort. I would like to thank Gary Langer of ABC News, Kathleen Francovic of CBS News, David Moore and Jeff Jones of the Gallup Organization, Dana Blanton of FOX News, and John Gorman of Opinion Dynamics Corporation for their help. Larry Hugick May 2004

8 Presidential Approval 8 Methodology Data This research study uses data from national polls conducted by ABC News/Washington Post; CBS News; CNN/USA Today/Gallup; FOX News/Opinion Dynamics; and Newsweek. 3 Presidential approval ratings from these five organizations were gathered for the period 1997 to Any data from polls conducted entirely in one day were excluded. All polls were conducted by telephone among the 18 and older general population except for FOX News polls which are limited to registered voters. The principal sources of approval data for this study are the PollingReport.com web site and the University of Connecticut s Roper Center ipoll databank. Historical Gallup inperson approval ratings for are taken from Presidential Approval: A Sourcebook, by George C. Edwards III, with Alec M. Gallup. Spreadsheets were created to analyze data by each organization and include the following information: Sample size and description Dates of interviewing Percent approve, percent disapprove, and percent no opinion (i.e. don t know or refused responses) Quarterly averages calculated from individual polls within each quarter Yearly averages calculated from quarterly averages within each year PSRAI requested each of the organizations included in the study to provide some details about polling methods, including question placement, weighting procedures, respondent selection, handling of a don t know response, callback procedures, and 3 Some polls included in this study as ABC News/Washington Post were done independently by either organization and not together with the partner. Some polls included as CNN/USA Today/Gallup polls were actually conducted independently by Gallup. Some polls identified as CBS News were done jointly with The New York Times, while others were done independently.

9 Presidential Approval 9 sample type. A detailed summary of responses can be found in the technical appendix section of this report. Computing Quarterly and Yearly Averages Quarterly and yearly averages are used for most of the analysis in this report. Quarterly averages were computed by averaging approval, disapproval, and no opinion ratings for individual polls within each quarter. 4 Yearly averages were then calculated using the quarterly averages within each year. Tables in the Analysis and Results section of this report summarize yearly averages for each president. Line charts using quarterly averages and scatter plots using individual poll results were also created to help with analysis. All tables and charts appear in the Analysis and Results section or technical appendix at the end of this report. Statistical Analyses An analysis of variance with post-hoc Bonferroni comparisons was run on quarterly averaged results in order to test for differences in approval, disapproval and DK ratings among the five organizations. To test for associations between DK level and approval/disapproval ratings, linear regressions were run based on individual poll data. These analyses are discussed in more detail in the Analysis and Results section. 4 Detailed tables of quarterly averages for can be found in the technical appendix.

10 Presidential Approval 10 Analysis and Results DIFFERENCES IN POLLING METHODOLOGY The following is a summary of the differences in methodology of the five polls studied, based on information provided directly from the researchers who conduct the polls 5 and information the various polling organizations make available through the Internet and other sources. A review of the methods of five media polls shows FOX, the new kid on the block, differing most in its procedures. Interviewing Methods and Practices FOX and Newsweek have the quickest turnaround time and shortest field periods two days. The other three polls tend to have longer field periods although CNN and CBS, in particular, conduct a fair number of two-day polls. All five polls make callbacks to non-contacts and ABC, CBS, and Newsweek also make refusal conversion attempts as a general rule. As a general rule, CNN makes refusal conversions only for polls in the field more than three days; FOX does not attempt refusal conversions. Only one poll CBS is conducted in-house by the media organization sponsoring the poll, but the other four polls rely on one field house, or limit their use of a different field house to rare occasions. Sampling and Weighting All five polls use RDD samples for their telephone polls rather than listed sample. FOX, which reports results based on registered voters instead of the total 18+ population, also differs from the other four polls in the way it deals with demographic variation in its sample. Opinion Dynamics weights the FOX poll sample only by race and ethnicity, and reports using sample controls to achieve demographic balance by gender, age, and region. 5 The technical appendix of this paper includes the responses of those conducting the five media polls to a short survey about their methods and practices.

11 Presidential Approval 11 Apparently, aside from its race weighting, FOX pollsters take no specific steps to deal with the tendency of unweighted samples to include too many better educated respondents and too few respondents with low levels of formal education. The other four polls take a more conventional approach to demographic weighting. They all weight their obtained sample by gender, race/ethnicity, age, and education. CBS, CNN and Newsweek also weight by geographic region. ABC does not specifically weight by region, but achieves geographic balance through other means. CBS also weights by the number of adults and the number of phone lines in the household. CNN also weights by the number of phone lines, but not by the number of adults. Interestingly, none of the five media polls is routinely weighted by party identification, a technique commonly used in polls done for political candidates by partisan pollsters. Typical sample sizes for the five media polls are quite similar. CNN and Newsweek usually poll about 1,000 adults; although Newsweek s standard sample size in polls with approval ratings was 750 adults prior to ABC s typical sample size is 1,200 adults, and FOX polls generally have a sample of about 900 registered voters. CBS s sample size varies from poll to poll, ranging from about 700 to 1,200 adults. Respondent Selection A variety of methods are used by the five media polls to select a respondent within household. Newsweek asks to speak with the youngest male/oldest female who is at home, a method CNN also used until late CNN now uses the most recent birthday method, as does FOX. ABC uses a variation of the most recent birthday method adjusted so interviewers ask to speak with a male 75% of the time and a female 25% of the time. CBS has the most rigorous approach, first determining the number of

12 Presidential Approval 12 adults in the household and selecting one at random, whether or not the person is at home, and making an appointment, if necessary, to interview the selected respondent. Questionnaire Issues Four of the five media polls in our study report placing the approval question very high in the questionnaire. CBS, CNN, and Newsweek report putting the approval ratings question first, except in election years, when trial heats or other election-related questions may take precedence. ABC reports asking the approval question first almost always. FOX is the one poll that does not have a standard policy of placing the approval question at the front of the interview, reporting that placement varies from poll to poll. Specific interviewer instructions about whether to probe for a response among those who do not say they approve or disapprove vary from poll to poll. CBS and CNN indicate that no probing is done, and such responses are simply recorded as don t know. Interviewers working on the Newsweek poll are specifically instructed to limit probing to cases where a respondent say it depends. ABC and FOX report using a full probe, re-asking the approval question of all respondents who do not say they approve or disapprove. Changes in Methodology during the Study Period Representatives of the polling organizations were asked if any major methodological changes were made to their polls during the period of this study, 1997 to ABC reports no major changes. The four other polls report some changes, none of which seems truly major. CBS reports adding callbacks to not in-service numbers in the past two years. As noted earlier, CNN changed to the most recent birthday respondent selection method in late FOX reports experimenting with different CATI systems that vary respondent selection or callback procedures. The Newsweek poll increased its sample size from 750 to 1,000 in January 2001.

13 Presidential Approval 13 TREND LINE FOR ALL POLLS Now turning to the polling data collected over this the seven-year period of our study, Chart 1 displays the key statistic (percent approve) for every individual poll conducted by the five organizations. 6 During Bill Clinton s second term in office, approval ratings generally moved within a 20-point range. For the first year of his second term (1997), Clinton received approval ratings in the mid-50s to low 60s. In late January 1998, Clinton approval jumped to around 70% as the public rallied around him shortly after the Monica Lewinsky scandal broke. Clinton approval then fell back, but rose again to the high 60s/early 70s around the time the House voted for impeachment in late December 1998 and his acquittal in February For the rest of Clinton s presidency, his ratings bounced around the 60% mark. Approval ratings of George W. Bush during his first three years in office followed a very different pattern. Prior to September 2001, Bush approval moved within the 50% to 60% range. After the 9/11 terrorist attacks, Bush approval jumped to over 80% in all polls, affected by the largest rally-round-the-flag effect on record. 7 As the rally faded gradually, Bush approval followed a steady downward path through early 2003, when it jumped upward in another rally effect from the Iraq war. It then resumed its downward trajectory until the end of 2003 when Saddam Hussein s capture produced yet another jump in Bush approval. With a few exceptions, Bush approval in polls taken from 2001 to 2003 ranged between 50% and 90%. Chart 1 makes it clear that presidential approval ratings in recent years did not follow a straight line for very long. Such movement makes it very difficult for us to 6 As noted previously, one-night polls by these organizations are excluded.

14 Presidential Approval 14 compare and contrast the results of polling conducted by the five organizations based on individual-level data. Since the five organizations tended to poll at different times and approval ratings were rarely static, timing is such a powerful factor that it tends to obscure any differences that might result from variation in research methods and practices. Moreover, the sample size of a typical individual poll is no more than about 1,000; meaning that the difference between two approval ratings must be more than six percentage points to meet the criteria for statistical significance. Nonetheless, an examination of the individual poll data in Chart 1 suggests that, although the five polls follow the same general trend line, there are certain patterns in the results of particular polls that do not appear to be the result of random variation. In the last two years of Clinton s presidency, the FOX approval ratings appear to be consistently higher than those of other organizations, and the Newsweek results appear to be consistently lower. During the Bush years, ABC approval ratings seem to be consistently on the high side and the Newsweek numbers consistently on the low side. 7 The 9/11 rally event and its impact on Bush approval is detailed in a paper by Larry Hugick and Margie Engle of Princeton Survey Research entitled Rally Events and Presidential Approval: An Update presented at the 2003 AAPOR conference in Nashville, Tennessee.

15 Chart 1: Presidential Approval Across Organizations Clinton Years Bush Years /11 attacks 84%-92% % Approve SOU Address Jan 27, %-70% House approves impeachment Late Dec 98 67%-73% Iraq War 67%-73% Saddam s Capture 54%-63% Lewinsky scandal goes public late Jan 98 54%-59% 40 Jan 97 Jan 98 Jan 99 Jan 00 Jan 01 Jan 02 Jan 03 Jan 04 Month and Year CBS ABC CNN FOX Newsweek

16 PATTERNS OBSERVED IN YEARLY AVERAGES In order to reduce sample size limitations and minimize the impact of timing in comparing poll results, we combined individual polls to calculate quarterly and yearly averages in the percent approve, percent disapprove, and percent don t know for each of the five organizations. Table 1, which displays yearly average ratings for George W. Bush from 2001 to 2003, reduces the data for this period to 15 sets of figures. These yearly averages confirm the impressions derived from the trend line. Overall, the five polls approval ratings are quite similar, but certain organizations display a tendency toward higher or lower ratings. The range in yearly averages for Bush approval across all five polls was only four to five points for these three years, but in two of the three years (2002 and 2003) ABC s rating was highest and Newsweek s the lowest. Oddly, 2001 was an exception with Newsweek s average approval at the high end of the scale. Further examination, however, revealed that this departure from the typical pattern is an artifact of the uneven distribution of Newsweek polls in the third quarter of Newsweek did no polls that quarter before the 9/11 terrorist attacks, and therefore its third quarter average was skewed by the extraordinary rise in Bush approval after 9/11 (85%). When the third quarter data is removed from the calculation of yearly averages for 2001, the typical pattern re-emerges, with ABC having the highest average approval (68%) and Newsweek the lowest (65%). These yearly averages for George W. Bush also show at least as much variation in the percent disapprove as the percent approve, between 5 and 6 points for the three years. While ABC ratings tend to be the highest or second highest, FOX, not Newsweek tends to be lowest. The percent don t know also shows some consistency in the yearly averages, with ABC tending to be lowest and FOX the highest.

17 Presidential Approval 17 Table 1 George W. Bush Job Ratings: 2003 % Approve % Disapprove % No opinion ABC CNN CBS FOX NW George W. Bush Job Ratings: 2002 % Approve % Disapprove % No opinion ABC CNN FOX CBS NW George W. Bush Job Ratings: 2001 % Approve % Disapprove % No opinion NW ABC CBS CNN FOX Note: FOX News figures based on registered voters. Now looking at yearly averages for the four years of Bill Clinton s second term, the data has some gaps, and does not show all the same patterns as the Bush data. FOX did not begin asking the standard approval question until 1998 and therefore no data is available for With the exception of one poll, Newsweek did not ask the Clinton approval question on polls conducted during election year 2000, and is therefore not represented in the yearly averages. Consistent with the patterns observed in yearly approval averages during the Bush years, Newsweek has the lowest approval rating in the three years it was active. However, during the Clinton years it was FOX not ABC

18 Presidential Approval 18 that generally had the highest approval ratings. ABC had the highest average disapproval during Clinton s second term. Patterns in don t know are also somewhat different from the Bush years. ABC s average ratings tend to be the lowest, but it is Newsweek and CBS not FOX that have the highest average don t know. Table 2 Bill Clinton Job Ratings: 2000 % Approve % Disapprove % No opinion FOX CBS ABC CNN NW n/a n/a n/a Bill Clinton Job Ratings: 1999 % Approve % Disapprove % No opinion FOX CNN ABC CBS NW Bill Clinton Job Ratings: 1998 % Approve % Disapprove % No opinion FOX CBS CNN ABC NW Bill Clinton Job Ratings: 1997 % Approve % Disapprove % No opinion ABC CBS CNN NW FOX n/a n/a n/a Note: FOX News figures based on registered voters.

19 Presidential Approval 19 STATISTICAL ANALYSIS OF QUARTERLY AVERAGES To put differences among the five media polls to a statistical test, Jonathan Best used the analysis of variance (ANOVA) technique to conduct a series of analyses with a subset of the data base created for this study. The data used for these analyses were limited because some polls were not active in certain years and polling done at certain times was atypical or distorted by external events. A description of the procedures and findings follows. ANOVA Procedures and Findings In order to compare ratings from the five polling organizations, one-way ANOVAs were run using quarterly averages for percent approve, percent disapprove, and percent don t know/refused as the dependent variables. Only quarters where every organization polled were included. All pre-september 11 results were excluded from the Bush data. This is because early approval ratings are atypical because of their higher levels of DK/Refused, and data for the third quarter of 2001 were distorted by the huge rally effect observed after the terrorist attacks on New York and Washington. Quarters included in analyses are 1st quarter 1998 through 4th quarter 1999 for Clinton and 4th quarter 2001 through 4th quarter 2003 for Bush. Results are summarized in Table A. TABLE A Sum of Squares df Mean Square F Sig Approve Between Groups Within Groups Disapprove Between Groups Within Groups DK/Refused Between Groups Within Groups

20 Presidential Approval 20 Analysis shows no significant difference in approval or disapproval ratings by polling organization. However, there is a significant difference in levels of DK/Refused. 8 Running post-hoc Bonferroni comparisons shows that ABC and CNN get consistently lower DK/Refused levels than the three other organizations. On average, ABC gets over 4 percent lower DK/Refused responses than CBS, FOX and Newsweek. CNN s DK/Refused ratings are almost 3 percentage points lower than CBS and more than 3 percentage points lower than both FOX and Newsweek. CBS, FOX and Newsweek average DK/Refused levels are all about the same. TABLE B Org A Org B Mean Difference (A-B) Sig. ABC CNN CBS -4.1*.000 FOX -4.8*.000 NW -4.9*.000 CNN ABC CBS -2.9*.000 FOX -3.6*.000 NW -3.7* When we run the analysis of variance for each president separately we get the same results no significant differences in approval or disapproval ratings, but a difference in DK/Refused. For this analysis by president, we dropped quarters where an organization did only one poll (2 nd Q, 1998; 3 rd Q, 1999; 4 th Q, 1999).

21 Presidential Approval 21 VISUAL REPRESENTATION OF POLL DIFFERENCES Although not large enough to meet the criteria for statistical significance in the ANOVA, some consistent differences from quarter to quarter are seen in average approval ratings for the five media polls. Chart 2 shows graphically how the various polls tracked with one another over the five years that were used for the ANOVA. As shown in Chart 2, almost without exception, Newsweek s approval ratings tend to be the lowest for both presidents. 9 During the Bush years at least, CBS tracks pretty consistently as second lowest to Newsweek. ABC tends to be at the opposite extreme during the Bush years, with the highest approval ratings. Chart 2 Presidential Approval: Selected Quarters Clinton Years Bush Years % Approve Year/Quarter ABC CBS CNN FOX NW 9 Newsweek data for second quarter 1998 and third quarter 1999 and ABC News data for third quarter 1999 and fourth quarter 1999 are not shown in charts containing quarterly averages because of low activity (only one poll in each quarter).

22 Presidential Approval 22 While Newsweek tends toward a lower percent approve, it tends to be in the middle as far as percent disapprove. As shown in Chart 3, ABC and FOX are most likely to be at the extremes as far as disapproval. ABC tends to register the highest disapproval; FOX the lowest. Again, although these differences are not large enough to be statistically significant in the ANOVA, their consistency suggests they bear further study. Chart 3 Presidential Disapproval: Selected Quarters Clinton Years Bush Years % Disapprove Year/Quarter ABC CBS CNN FOX NW Moving to differences among the five polls in the DK/Refused ratings which, in fact, are statistically significant, Chart 4 shows the clear separation between the high DK and low DK polls. Leaving FOX aside for the moment, from quarter to quarter the DK level for ABC and CNN is always lower than the DK level for CBS and Newsweek. ABC tends to be lowest; Newsweek the highest. While FOX appears to be a high DK poll consistently different from ABC and CNN, the level of DK in the FOX poll seems different in the Bush data than it is in the Clinton data. During most of the Bush years, FOX had the highest DK level, often reaching 10% or higher. In contrast,

23 Presidential Approval 23 FOX was never higher than both CBS and Newsweek in quarterly averages for DK levels in the Clinton data. Chart 4 Presidential No Opinion Ratings: Selected Quarters Clinton Years Bush Years % Don' t Know Year/Quarter ABC CBS CNN FOX NW In addition to analyzing the quarterly averages for approval and disapproval of the five polling organizations, we created an approval ratio which removes the DK from the denominator. Specifically, the approval ratio is defined as follows: Approval Ratio = Approval (Approval + Disapproval) Differences in the approval ratio among the five polls over time are displayed in Chart 5. At least for the most recent years studied, the four polls with a conventional methodology ABC, CBS, CNN, and Newsweek move even closer together when the approval ratio replaces basic approval as the statistic used for evaluation. The FOX results, however, do not move toward the others, and follow a somewhat different path. In its approval ratio, FOX tends to be higher than the other four polls for most of the selected quarters

24 Presidential Approval 24 analyzed Bush post-9/11 and Clinton during the middle two years of his second term, when all five polls were active. Chart 5 Approval Ratio: Selected Quarters Clinton Years Bush Years % Approve Year/Quarter ABC CBS CNN FOX NW INVESTIGATING THE DON T KNOW FACTOR If the levels of don t know is the primary difference in the results of the five media polls the only difference the ANOVA found to be statistically significant what is responsible for the DK factor? Question order and interviewer handling of the approval question would seem to be the most likely source of this difference. Leaving aside FOX to focus on the four polls that use a conventional telephone survey methodology, question order can be removed as a likely factor. All four polls place the approval question high in the interview, generally the first substantive question except for trial heats and other election-related questions. The extent to which the various polling organizations instruct interviewers to probe DKs to the approval question might explain the differences between three of the polls ABC, CBS and Newsweek. ABC, which gets

25 Presidential Approval 25 the lowest level of DKs, has its interviewers re-ask the approval question of all respondents who do not answer approve or disapprove the first time around. The practice is not shared by CBS and Newsweek, the two conventional polls that typically get the highest levels of DK. CBS interviewers do not probe at all, while Newsweek interviewers do only limited probing, when a respondent offers a mixed opinion, such as it depends. The level of difference in DKs between these two organizations and ABC, about 4 percentage points, is consistent with typical differences observed between polls that ask trial heats with a lean question vs. polls that do not ask a lean question. This does not explain, however, why the other low DK poll CNN consistently gets a level of DK almost as low as ABC. Gallup researchers who conduct the CNN poll say their interviewers do not probe the DKs at all. The interviewer probing explanation also does not work for the FOX poll, which during the Bush years has registered the highest levels of DK. In fact, FOX is the only poll beside ABC that reports routinely probing all DKs to the approval question. What s more, FOX is the only poll of the five that does not report always placing the approval question very high in the interview. All other things being equal, an approval rating asked later in the interview would be expected to register a lower level of DK, not a higher level. 10 In order to see what effect reducing the level of DK has on presidential approval and disapproval, Jonathan Best conducted the following analyses using a subset of the study data base. Selecting all polls where the percent DK was 15% or less (n=618), approval ratings were plotted on a graph by DK (see Chart 6), and a linear regression line was fit through these data. 10 Crespi, Irving. Pre-election Polling: Sources of Accuracy & Error (Russell Sage Foundation: 1988), page 104.

26 Presidential Approval 26 Chart 6 Approval by Don't Know: Selected Polls % Approve % Don't Know % Approve Linear (% Approve) A clear relationship between the level of approval and the level of DK is found. In fact, this regression line has a significant non-zero slope of This means that as DK increases by one percentage point, approval decreases by percent. Conversely, as DK decreases, as it does with probing, approval increases at the same rate approximately eight-tenths of a point for each one-point drop in the DK. This same relationship is not found between disapproval and DK. The regression line fit to these data has a slope (-0.176) that is not significantly different than zero. In other words, as DK changes, disapproval ratings tend to remain about the same. (See Chart 7).

27 Presidential Approval 27 Chart 7 50 Disapproval by Don't Know: Selected Polls % Disapprove % Don't Know % Disapprove Linear (% Disapprove) To ensure that the elevated Bush approval ratings during the post 9/11 period did not distort the data, these analyses were also done separately for each president. When the analysis is re-run only for the Bush data, similar results are found to those for the larger data set. There is a significant negative relationship between DK and approval, and no significant relationship between DK and disapproval. When the analysis is re-run for the Clinton ratings, the same relationship is found between approval and DK, but there is also a small, but statistically significant, relationship between DK and disapproval. For Clinton s ratings, as DK decreases by one percentage point, approval increases by.78 points and disapproval increases by.22 points.

28 Presidential Approval 28 TABLE C: RELATIONSHIP BETWEEN DK AND APPROVAL/ DISAPPROVAL RATINGS Beta Sig Both Approve * Disapprove Bush Approve * Disapprove Clinton Approve * Disapprove * These findings give support to the hypothesis that probing and interviewing effects are the reason for the DK factor. The tendency for lower don t knows to push the approval level up while leaving disapproval largely unaffected is consistent with a positivity bias frequently observed in response to survey questions about low salience issues. When people don t have especially strong opinions about something, pushing them for a response tends to elicit positive rather than negative views. But as noted previously, attributing all these differences to probing does not square with reports about interviewer practices at CNN and FOX. CNN, whose interviewers are not given specific instructions to probe, nonetheless gets low DKs; FOX, whose interviewers are instructed to probe, gets high DKs. It may be that less obvious house effects play an important role. Perhaps in some cases differences created by interviewer characteristics, general interviewer training, or habits formed from administering other surveys in the same phone room overpower any specific procedures in place for handling the DKs to the presidential approval question. One final piece of analysis in evaluating the DK factor was to compare and contrast the results of the five polls when approval ratings were well outside the normal

29 Presidential Approval 29 range in which they travel. Table 3 shows the figures for each of the five polls when they got their highest approval rating for George W. Bush in the fall of It is noteworthy that ABC, with its full probing of the DKs had only one percent DK in their high watermark poll and the highest approval rating of the five (92%). In these atypical circumstances, CNN is indistinguishable from the three high DK polls in the percent DK. FOX again shows itself to deviate from the other polls in an unpredictable manner. While the four conventional polls hit their high point for Bush approval in late September to early October, FOX did not reach its peak until mid-november. When other polls registered approval at or near the 90% mark, the FOX poll had approval in the 80% to 84% range. Table 3 HIGHEST BUSH APPROVAL RATING FOR FIVE POLLS FALL 2001 Approve Disapprove DK Dates of Interviewing % % % ABC /8-9/01 CNN /21-22/01 CBS /20-23/01 FOX /14-15/01 Newsweek /11-12/ Ratings in a 10/18-19/01 Newsweek Poll were the same.

30 Presidential Approval 30 COMPARISONS WITH GALLUP IN-PERSON APPROVAL RATINGS One last area examined for this study is whether current presidential approval ratings in contemporary telephone polls can be compared directly with approval ratings from past presidents, as measured by The Gallup Poll s in-person survey methodology. Since telephone methodology did not come into its own until the late 1970s/early 1980s, the in-person data is the only polling data available for presidents before Jimmy Carter. And since Gallup switched to telephone surveys for its reporting of presidential approval ratings beginning with President George H.W. Bush, in recent years in-person poll data has not been available for direct comparisons with telephone poll results. There is one period, however, when there was sufficient in-person Gallup Poll data and telephone survey data to allow for good comparisons the 1980s during Ronald Reagan s presidency. The CBS News poll is the obvious choice and perhaps even the only choice for making these comparisons. CBS is the only poll of the five studied that had the following combination of characteristics: 1) was sufficiently active during the Reagan years; 2) used the same basic methodology then as it does today 12 ; 3) did not change phone rooms; and 4) had continuity of leadership. To make the comparisons between CBS telephone poll approval ratings and those from Gallup in-person surveys during the Reagan years, the data for each organization were combined to create yearly averages. 13 (The frequency of polling was not as high at that time and therefore quarterly averages were not an option.) The yearly averages were then combined to calculate grand average yearly differences for percent approve, percent 12 In an to Larry Hugick on 4/23/04, Kathleen Francovic indicated that CBS used the same basic methodology in the 1980s, except for weighting by the number of telephone lines in the household, not added until the tail end of the Reagan years. 13 The fourth quarter of 1986 was excluded from the calculation for that year because of distorting effect of the Iran-Contra scandal.

31 Presidential Approval 31 disapprove, and percent DK. As seen in Table D, CBS telephone polls track extremely close to Gallup Poll in-person results in the grand averages for Reagan approval, not varying by more than two points on any of the three measures. TABLE D: CBS TELEPHONE VS. GALLUP IN-PERSON RESULTS: Approve Disapprove DK % % % 1981 CBS Gallup Difference CBS Gallup Difference CBS Gallup Difference CBS Gallup Difference CBS Gallup Difference CBS Gallup Difference CBS Gallup Difference CBS Gallup Difference Average Difference 1-2 2

32 Presidential Approval 32 This analysis shows us how CBS compared with the Gallup in-person results in the past and, inferentially, how it might compare today. But what about the other four polls? To arrive at rough estimates for how figures from ABC, CNN, FOX, and Newsweek might compare to Gallup in-person results, two steps were required: 1) calculating average differences between CBS approval figures and each of the other four polls; and 2) applying key provided by the CBS vs. Gallup differences to extrapolate how much the other four polls differ from the in-person results. For the comparisons to CBS, we decided to use the most recent data possible, calculating average differences between each poll and CBS for each of the three Bush years, and then averaging the yearly differences. (The 2001 average excluded the problematic third quarter.). Table E shows step one of this analysis. TABLE E: ANNUAL DIFFERENCES BETWEEN CBS AND OTHER ORGANIZATIONS (PERCENTAGE POINTS) Approve Disapprove DK 2001 ABC CNN FOX Newsweek ABC CNN FOX Newsweek ABC CNN FOX Newsweek Average ABC CNN FOX Newsweek

33 Presidential Approval 33 The differences for each poll from CBS during the George W. Bush years were then adjusted to take into account CBS differences from Gallup in-person results during the Reagan years. Table F summarizes the final results of the two-step process, calculations for how each approval figure for each of the five contemporary telephone polls compares with those from historic in-person Gallup polls. These findings would indicate that all five telephone polls come reasonably close to approximating the approval ratings that had been obtained in the past using an in-person interviewing methodology. The high DK telephone polls, however, appear to be slightly closer to the in-person results than the low DK telephone polls on the key measure of percent approve. CBS, FOX, and Newsweek are all within a point in percent approve. ABC averages three points higher, and CNN averages two points higher than in-person results. It should be noted, however, when we go beyond this most widely-reported statistic, the story changes somewhat. FOX diverges from in-person results more than any other telephone poll in its average percent disapproval (-5) and average don t know (+4). ABC estimates of disapproval are a perfect match to in-person results, while CNN and Newsweek are just a point lower on this same statistic. TABLE F: ESTIMATED DIFFERENCE BETWEEN TELEPHONE POLL APPROVAL RATINGS AND GALLUP IN-PERSON RATINGS (PERCENTAGE POINTS) Approve Disapprove DK ABC CNN FOX CBS Gallup In-Person Newsweek

34 Presidential Approval 34 Several caveats are appropriate for the preceding analysis. First, it does not account for real world differences that might change the results of polls today compared with the 1980s. And it also assumes that the patterns seen in differences in poll ratings for one president (George W. Bush) would hold for a different president. Conclusions The five media polls studied track very closely with one another in their presidential job approval ratings, especially when the focus is narrowed to the mostwidely reported statistic (percent approve) rather than the full set of results that includes the percent disapprove and don t know. For the period of 1997 to 2003, the typical difference across all five polls in average annual approval is just four percentage points, and the five poll trend lines move up and down largely in concert. That said, our analysis does not suggest that presidential approval ratings for the five polls are truly interchangeable. There are some differences that follow a consistent pattern over time, and even one difference the level of don t know that is statistically significant. To put the approval results of these five polls in perspective, our study indicates they should be divided in two ways. First, it is useful to separate the four polls that employ a conventional telephone survey methodology with a full demographic weighting from the one poll that takes a different methodological approach. The ABC News/Washington Post poll, the CBS News poll, the CNN/USA Today/Gallup poll and the Newsweek poll all use a conventional methodology. While these four polls differ in some specific aspects of methodology such as length of a typical interviewing period and respondent selection procedures they have enough in common that in the few ways their results seem to differ, they do so in a predictable and consistent manner. The FOX

35 Presidential Approval 35 News/Opinion Dynamics poll, the newest of the five polls, differs from the other four in basic methodology, using a registered voter base and only limited demographic weighting of its sample. The FOX poll maintains demographic balance through sample controls rather than weighting, does not do much to control for respondents education levels, and does not have a general policy of placing the approval question at the front of the interview. The second important way to divide the five polls is by their typical level of don t know to the approval question. Two of the polls with a conventional methodology are low DK (ABC and CNN) and two are high DK (CBS and Newsweek). FOX has been a high DK poll, and has shown an average percent approval remarkably close to CBS and Newsweek in recent years. However, there are enough differences in FOX and these two polls on other measures to suggest it needs to be kept separate. Statistical analysis indicates that ABC and CNN consistently get a lower level of DK to the approval question than the other two conventional polls, CBS and Newsweek, and FOX. What s more, analysis suggests that lower DKs tend to drive approval ratings higher while leaving disapproval largely unaffected. For each one point reduction in the level of DK, approval rises by eight-tenths of a point but disapproval is statistically unchanged. All this suggests that poll watchers should exercise greater caution when interpreting approval ratings from polls in a different DK category. Newsweek approval ratings, for example, are more comparable to those of CBS than to those of ABC. The FOX/Opinion Dynamics poll has tracked remarkably close to the four conventional polls particularly those in the high DK category in percent approve during recent years. Nonetheless, analysis identifies differences below the surface in the