THE INFLUENCE OF VISUAL LAYOUT ON SCALAR QUESTIONS IN WEB SURVEYS LEAH MELANI CHRISTIAN

Transcription

1 THE INFLUENCE OF VISUAL LAYOUT ON SCALAR QUESTIONS IN WEB SURVEYS By LEAH MELANI CHRISTIAN A thesis submitted in partial fulfillment of the requirements for the degree of MASTER OF ARTS IN SOCIOLOGY WASHINGTON STATE UNIVERSITY Department of Sociology DECEMBER 2003

2 To the Faculty of Washington State University: The members of the Committee appointed to examine the thesis of LEAH MELANI CHRISTIAN find it satisfactory and recommend that it be accepted. Chair ii

3 ACKNOWLEDGMENTS I would like to thank Don Dillman for introducing me to this area of research and for allowing me the opportunity to help design the experimental comparisons tested in the Spring 2003 Student Experience Internet survey. I would also especially like to thank him for Chairing my master s thesis committee and his continued support of my work in survey methodology. Additionally, I would like to thank Thomas Rotolo, Nella Van Dyke and Gregory Hooks for their participation of my committee and their individual insights and encouragement. I would also like to acknowledge the financial support provided for this research by The WSU Agricultural Research Center under Western Region Project W-183, The Social and Economic Sciences Research Center, The National Science Foundation Division of Science Resource Statistics, the USDA-National Agricultural Statistics Service, and The Gallup Organization I would like to thank Thom Allen, study director, and Vincent Kok of the Social and Economic Sciences Research Center at Washington State University for their key efforts in implementing and collecting the data from the web survey. Two additional sociology graduate students working on the project, Michael Stern and Jolene Smyth, were very supportive and helpful in discussing and analyzing the data included in this thesis. I would especially like to thank my family for their continued support as I have pursued my education and goals throughout the years. My mother, Lauren Christian, provided consistent emotional support and served as my best editor. My father, Darrel Christian, was very supportive of my continued interest in education and has always helped in any way that I needed. Lastly, thanks to Kela Quinn my sister, best friend, and confidant throughout. iii

4 Most importantly, I would like to express gratitude to my life partner and husband, Eugene Macintyre for his support and encouragement throughout my pursuit of this master s degree. His unyielding belief in me has been a constant source of energy. Eugene s strength and love have continued to inspire me throughout this process, even from afar. Bucky and Rabun also provided constant love and support. iv

5 THE INFLUENCE OF VISUAL LAYOUT ON SCALAR QUESTIONS IN WEB SURVEYS Abstract by Leah Melani Christian Washington State University December 2003 Chair: Don A. Dillman Ordinal scale questions are frequently used by sociologists. This thesis examines how the visual presentation and layout of response choices influence answers to ordinal scale questions in web surveys. This research extends previous experimentation on paper surveys to this new visual survey mode of the Internet to determine whether the results of varying scale layouts are similar or different. Two sets of comparisons were included in a web survey of a random sample of Washington State University students during the Spring 2003 Semester; 1591 completed surveys were submitted from the 3004 requested obtaining a response rate of 53%. One experiment included a response scale with all categories labeled and compared a vertical linear layout using 4-5 categories to three nonlinear layouts where categories were double or triplebanked in columns across the page. A second set of experiments compared a 5-point fully labeled scale to a polar point scale where the verbal labels were removed for the middle three categories to an answer box format where respondents reported a number in an answer space. Multiple replications within each experiment indicate significant differences in the means between formats as well as significantly different response distributions and these findings tend to confirm previous findings on paper questionnaires and suggest that respondents to both paper v

6 and web surveys are similarly affected by visual layout and presentation (Christian and Dillman, In Press). Strong evidence now exists, that the visual presentation and layout of response scales influences respondent answers to self-administered questionnaires and needs to be considered when designing surveys that use ordinal scale questions. These construction differences appear to be important in helping surveyors understand why responses may vary across modes. The increasing use of mixed-mode surveys suggests the need to understand how the mode of communication, visual or aural, can influence how surveyors design questions and how respondents answer those questions. vi

7 TABLE OF CONTENTS Page ACKNOWLEDGEMENTS iii ABSTRACT..... v LIST OF FIGURES. ix LIST OF TABLES...x LIST OF CHARTS.....xi SECTIONS: INTRODUCTION...1 THEORETICAL BACKGROUND....4 Responding to a Self-Administered Questionnaire.5 Visual Languages 7 Scalar Questions.10 EXPERIMENTAL COMPARISONS 16 Linear vs. Nonlinear Scale Layout Paper Survey Experimental Comparisons and Results...16 Web Experimental Comparisons 17 Fully Labeled vs. Polar Point vs. Answer Box..19 Paper Survey Experimental Comparisons and Results..19 Web Experimental Comparisons 20 METHODS FINDINGS..25 Linear vs. Nonlinear Scale Layout..25 vii

8 Fully Labeled vs. Polar Point vs. Answer Box 28 DISCUSSION..31 Linear vs. Nonlinear Scale Layout...31 Fully Labeled vs. Polar Point vs. Answer Box...33 CONCLUSIONS..34 FIGURES. 38 TABLES...43 CHARTS..48 REFERENCES.58 viii

9 LIST OF FIGURES 1. Figure 1: Linear vs. Nonlinear Experimental Comparisons A. 5 Category Scale. 38 B. 5 Category Scale. 38 C. 5 Category Scale with don t know quasi filter D. 4 Category Scale Figure 2: Fully Labeled vs. Polar Point vs. Answer Box Experimental Comparisons A. 5 Category Desirability Scale w/ Midpoint.40 B. 5 Category Satisfaction Scale w/ Midpoint.40 C. 5 Category Satisfaction Scale w/ Midpoint.41 D. 5 Category Outstanding to Terrible Scale w/ no Midpoint.41 E. 5 Category Satisfaction Scale w/ Midpoint and Don t Know Quasi Filter...42 F. 5 Category Satisfaction Scale w/ Midpoint and Don t Know Quasi Filter...42 ix

10 LIST OF TABLES 1. Table 1: Linear vs. Nonlinear Experimental Comparisons A. 5 Category Scale.. 43 B. 5 Category Scale.. 43 C. 5 Category Scale with don t know quasi filter..44 D. 4 Category Scale Table 2: Fully Labeled vs. Polar Point vs. Polar Point w/# s vs. Answer Box A. 5 Category Desirability Scale w/ Midpoint..45 B. 5 Category Satisfaction Scale w/ Midpoint..45 C. 5 Category Satisfaction Scale w/ Midpoint..46 D. 5 Category Outstanding to Terrible Scale w/ no Midpoint..46 E. 5 Category Satisfaction Scale w/ Midpoint and Don t Know Quasi Filter..47 F. 5 Category Satisfaction Scale w/ Midpoint and Don t Know Quasi Filter..47 x

11 LIST OF CHARTS 1. Chart 1: Linear vs. Nonlinear Experimental Comparisons A. 5 Category Scale. 48 B. 5 Category Scale. 49 C. 5 Category Scale with don t know quasi filter...50 D. 4 Category Scale Chart 2: Fully Labeled vs. Polar Point vs. Polar Point w/# s vs. Answer Box A. 5 Category Desirability Scale w/ Midpoint..52 B. 5 Category Satisfaction Scale w/ Midpoint..53 C. 5 Category Satisfaction Scale w/ Midpoint..54 D. 5 Category Outstanding to Terrible Scale w/ no Midpoint..55 E. 5 Category Satisfaction Scale w/ Midpoint and Don t Know Quasi Filter..56 F. 5 Category Satisfaction Scale w/ Midpoint and Don t Know Quasi Filter..57 xi

12 INTRODUCTION Sociologists and other social scientists frequently use scalar questions on surveys to determine the attitudes, opinions, and behaviors of respondents. Measurement for scalar questions emerges from the nature of the variable categories"; therefore, the numbers and labels assigned to the categories are very important in determining what level of measurement is being used (Singleton and Straits 1999, p.109). Ordinal measurement is one of the most frequently used levels of measurement since its use allows social scientists to measure gradations of opinion. In ordinal scales, the categories are ordered and respondents are asked to select where they fit along an implied continuum (e.g. agree to disagree, positive to negative). There is considerable variation in how ordinal scales are presented to respondents in selfadministered questionnaires. These differences in the visual layout of ordinal scales seem particularly important since respondents are asked to determine the order of categories before deciding where they fit along the continuum. In an ordinal scale, the categories can be presented in a single column or row, or in multiple columns or rows. In addition, word labels may be provided for all categories or only for the endpoint categories. Very little research has addressed how these different visual layouts of ordinal scales may influence respondents answers. Construction differences can have meaningful effects on responses to survey questions because respondents draw on multiple attributes of a self-administered questionnaire when forming answers to survey questions. Specifically, a self-administered questionnaire consists of verbal and nonverbal languages (Jenkins and Dillman 1997). The nonverbal languages include numeric, symbolic, and graphical languages that convey meaning in addition to the verbal language. Both visual and aural surveys use paralanguage to provide meaning to the respondent beyond the verbal language. In visual surveys, the graphical language acts like paralanguage and is the visual conduit 1

13 through which the verbal, numerical, and symbolic languages are transmitted (Redline and Dillman 2002). Further, the verbal and nonverbal languages can independently and jointly create meaning and influence how respondents navigate through the survey and influence answers to individual questions (see Redline, Dillman, Dajani, and Scaggs, In Press; Christian and Dillman, In Press). Respondents gain information from these four languages as part of the context of the survey questionnaire and often interpret them as an important source of meaning. It has been shown that manipulating these visual languages influences answers to ordinal scale questions in paper, self-administered surveys (Christian and Dillman, In Press, Schwarz et. al. 1998, Schwarz et. al. 1991, Smith 1993). Schwarz, et. al. (1991) show how numerical language can influence answers to ordinal scale questions. The research by Smith (1993) and experimental replication by Schwarz, et. al. (1998) show that different graphical layouts of scales can influence respondents' answers. Further, the experimental research by Christian and Dillman (In Press) shows that a linear layout of ordinal scales, where all categories are presented vertically, produces different answers from nonlinear scales where categories are presented horizontally in two or three columns across the page. On this same survey, we also found differences between fully labeled scales and polar point scales (where all categories appear in a linear column with check boxes but only the endpoints are verbally labeled). Polar point scales also produced different responses than an answer box format where respondents report the number corresponding to their answer and no response scale is provided. The recent increase in the use of web surveys suggests the need to understand how visual languages influence responses to questions on web surveys. Since both web and paper questionnaires rely on visual communication and are self-administered, it might be expected that responses would be similar across paper and web modes. However, the completion of web surveys 2

14 also involves different response actions from mail surveys. The respondent processes the information from a monitor or screen instead of paper and reports responses using a mouse and keyboard instead of a pencil or pen. Further, the web screen is greater in width than length, unlike common paper sizes (8 1/2 by 11 ), a feature that encourages landscape construction of web pages and surveys. It is unknown whether these differences in the actions required to complete web surveys influence the effects of visual languages on respondent answers. The experiments reported here extend our previous knowledge of the influences of visual presentations on ordinal scales by testing visual manipulations of multiple ordinal scale questions on a web survey. Don Dillman and I created four alternative visual layouts of each question to expand on our previous findings (Christian and Dillman, In Press) and to experimentally compare results across four different versions. I also compared these results to those found on the paper questionnaire to determine whether responses are similar across paper and web modes. One experiment compared the effects of three nonlinear versions to the linear version for multiple questions. The three nonlinear versions compare double and triple- banked formats that read horizontally to ones that read vertically. We also tested the influence of adding numbers to a vertical nonlinear format. The second set of experiments compared a fully labeled to a polar point version with numbers, a polar point version without numbers, and the answer box version, again using multiple replications. We designed these experiments to show how independent and combined manipulations of the verbal and nonverbal languages influence respondents answers to a web self-administered questionnaire. 3

15 THEORETICAL BACKGROUND Survey methodologists are currently able to choose from more modes than ever before. Interview-administered modes include face-to-face, telephone, and Interactive Voice Response (IVR). 1 Interview-administered modes rely primarily on aural communication; however, paper and Internet self-administered modes rely primarily on communication through both verbal and nonverbal visual languages. As the newer methods have developed (IVR and the Internet), they are used along with other modes and have not replaced existing modes. Mixed mode surveys are becoming more common, where the mode of surveying is changed during the survey process resulting in multiple modes being used for data collection. Mixed mode surveys can be used to decrease nonresponse and data collection costs. One concern with the use of mixed mode surveys is that survey questions are often constructed differently for the various modes. For example, a fully labeled scale may be used on a mail survey, but a polar point scale is more often used on a telephone survey to decrease the demand on the interviewer. As survey methodology is moving towards mixed-mode surveys, it is necessary to understand how these construction differences influence answers to survey questions and whether or how questions are comparable across modes. In particular, research by Dillman et. al. (2000) showed that answers to ordinal scale questions (that were polar point labeled) were similar across two aural modes (telephone and IVR) but differed from the answers given on two visual modes (mail and Internet). Respondents were more likely to give extreme responses, using the endpoints, on the aural modes (Dillman, Phelps, Tortora, Swift, Kohrell, and Berck. 2000). This research suggested that differences in responses might occur across aural and visual survey modes as respondents give more attention to internal categories on the visual modes. Additionally, 1 Interactive Voice Response is a particular telephone format where a prerecorded voice reads questions and the respondent answers using voice or touchtone entry 4

16 differences in responses may also occur within particular types of aural or visual modes. This thesis focuses on understanding differences within two self-administered modes, mail and Internet, and whether the visual languages used in these two modes may produce similarities or differences. The recent increase of web surveys and the use of Internet surveying, especially in certain populations where Internet access is provided such as surveys of employees and college students, highlights the need to begin to understand how respondents complete web surveys and to see how respondent answers compare with those collected using other modes. Internet surveys are limited in that coverage can only be assumed within certain limited populations; however, their popularity has increased because of several key advantages. Two important advantages of Internet surveys are the reduced costs of data collection and processing and the increased speed in which data are available for analysis. In addition, paradata (Heerwegh 2002) can be collected for all respondents that records how the respondent interacts with the web survey. Paradata includes information about how much time the respondent spent on each individual question and answer, whether certain formats encouraged respondents to change answers, and what changes were made. This thesis focuses on understanding how manipulating the visual languages used to construct ordinal scale questions influences answers to a web self-administered questionnaire and discusses how respondent answers might compare to those found using a paper self-administered questionnaire. Responding to a Self-Administered Questionnaire Tourangeau (1984) has summarized the process of how participants respond to interviewadministered surveys in four steps: 1) comprehending the question, 2) recalling relevant 5

17 information, 3) making a judgment, and 4) reporting a response 2. This model has been elaborated for self-administered questionnaires to include an initial step of perceiving the question stimulus and to specify that respondents must perceive and comprehend both the verbal and nonverbal languages of the questionnaire (Jenkins and Dillman 1997, Redline and Dillman 2002). It is important to note that verbal and nonverbal languages influence respondent behavior throughout the question/answer process in self-administered questionnaires. It has been argued that responding to surveys is governed by the conduct of conversation that is used in everyday life; therefore, people respond to surveys following the tacit assumptions that underlie the conduct of conversation 3 (Schwarz 1996; Grice 1975, 1978). These assumptions can be expressed as four maxims: the maxim of manner (contributions can be understood by the intended audience), the maxim of relation (contributions should be relevant), the maxim of quantity (contributions should be informative but not more so than required), and the maxim of quality (contributions should be truthful). These maxims combine to form a cooperative principle of conversation whereby ones contribution should be clear, informative, relevant, and truthful (Schwarz 1996, p.11). In self-administered surveys, the survey instrument represents the researcher s half of the conversation, and respondents assume the researcher is following this cooperative principle when designing the questionnaire. However, Schwarz (1996) says that researchers often violate these four maxims when designing survey questionnaires. The survey instrument is especially important because the conversation in a selfadministered survey is somewhat different from an interview conversation where information can 2 Other models discuss slight variations of these four steps but these four steps are generally agreed upon. 3 See Schwarz (1996) for an elaborate discussion of the tacit assumptions and logic of conversation based on Grice s (1975, 1978) outline of the logic of conversation and Levinson s (1983) summary of Grice s theorizing. 6

18 be continuously exchanged. The survey itself is the only information respondents have when completing the steps in the question-answer process. Since the survey instrument represents the researcher s half of the conversation, the respondent draws on the material presented in the questionnaire and assumes much of it is relevant to the conversation at hand; namely, completing the questionnaire. Since respondents assume the material provided in the questionnaire is relevant to the survey conversation, respondents systematically gain information from the context of the questions, including the formal features of the questionnaire, to understand what the researcher is asking. To disambiguate its meaning respondents turn to the context of the question, much as they would in daily life (Schwarz, 1996, p.48). In a research situation, the relevant context includes the formal features of the questionnaire including verbal, numeric, symbolic, and graphical languages. The respondent then makes systematic use of a wide range of questionnaire features to arrive at an interpretation (Schwarz, 1996). Thus, when responding to ordinal scale questions, I expect that respondents draw information from how the query is presented, the verbal and numeric language of the labels, and the graphical layout of the scale. Visual languages A theory of self-administered questionnaire and visual design is still in its infancy, but a conceptual framework for understanding visual influences in self-administered questionnaires has been developed elsewhere (Jenkins and Dillman 1997). Visual design theory applies ideas from research on cognition and visual perception (Glass and Holyoak 1986, Crick 1994, and Palmer 1999) to understand how respondents process and complete self-administered questionnaires 4. This literature suggests that respondents use fundamental processes of cognition and perception when 4 Jenkins and Dillman (1997) provide a detailed discussion of cognition and visual perception processes and how they apply to respondents answering self-administered questionnaires. 7

19 completing questionnaires suggesting that how information is presented to respondents will influence how it is processed (Jenkins and Dillman 1997). Thus, these concepts have been used to develop an understanding of how verbal and nonverbal languages 5 are important in understanding how respondents complete self-administered questionnaires. Verbal languages, the words used in communication, are important in all survey modes. As such, many research studies in survey methodology have focused on how to carefully choose question wording to convey the intended meaning (Schuman and Presser 1981, Payne 1951, Sudman and Bradburn 1974). Much of this research has focused on interview-administered questionnaires where information is presented aurally, but verbal language has also been discussed in relation to self-administered questionnaires (Schwarz, et. al. 1985; Schwarz, et. al. 1991b). In scalar questions, the verbal language used in the query and the response options are important sources of information for the respondent. Even though considerable research exists regarding the importance of understanding verbal language, there has traditionally been much less focus on understanding how nonverbal language influences answers to questions in self-administered questionnaires (see Wright and Barnard 1975, 1978). Nonverbal language includes numerical, symbolic, and graphical information, which may independently convey meaning separately from the verbal language or may influence the meaning that the verbal language portrays to respondents. Numerical language consists of the numbers used in a questionnaire such as question numbers which independently help respondents navigate through a questionnaire or numeric labels given to scale categories which may independently and/or jointly with verbal labels define the meaning of the categories in an ordinal scale question. Symbolic language consists of culturally agreed upon signs and symbols such as arrows used in 5 See Jenkins and Dillman (1997) for a discussion of why both verbal and spatial arrangement (which they call nonverbal) should be considered language. 8

20 directing respondents to a subsequent question. Finally, graphical language consists of such features as size, shape, location, brightness, and color. Graphical paralanguage 6 are visual elements that accompany the verbal, numerical, and symbolic languages and contribute to communication in self-administered questionnaires (Redline and Dillman 2002). The graphical language serves as the visual conduit because the verbal, numeric, and symbolic languages do not stand alone and can only be transmitted through the visual channel via graphic paralanguage (Redline and Dillman, 2002, p.181). Two experimental tests used branching instructions to demonstrate that the four visual languages influence how respondents navigate through a self-administered questionnaire. The experimental research indicates that manipulating several aspects of these four languages simultaneously (i.e. font size and brightness of the branching instruction, use of directional arrows, graphical spacing, and additional verbal language) increases compliance with branching instructions. One experiment surveyed university students, and 1266 students completed one of three versions of a questionnaire. The newly designed questionnaire versions 7 (two of the three that students received), where the visual languages were used in an effort to improve compliance with branching instructions, reduced commission errors (failure to skip when directed) from 20% to between 7-9% (Redline and Dillman 2002). The second experiment used the 2000 Decennial 6 The term paralanguage is used to refer to features that accompany speech and contribute to communication, e.g. loudness and tempo and facial expressions and gestures (Redline and Dillman 2002, p.181). Graphical paralanguage is used because visual elements can be thought of similarly; that is, they accompany text and contribute to communication but are not language per se (Redline and Dillman 2002, p.181). 7 Both a prevention method, designed to prevent errors associated with branching instructions, and a detection method, designed to help respondents detect their errors after they have been made, were developed combing multiple manipulations of the visual languages. 9

21 United States Census to test similar designs 8 intended to manipulate the visual languages in an effort to increase compliance, and commission errors were reduced from 21% to 13-15% (Redline, Dillman, Dajani, and Scaggs, In Press). These two experiments manipulated multiple aspects of the four visual languages simultaneously; therefore, they were unable to disentangle the individual effects of independent changes. Dillman conducted a survey using two versions of a mail self-administered questionnaire given to university students where multiple independent and one combined manipulation of symbolic and graphical languages significantly influenced respondent answers (Christian and Dillman, In Press). In these independent manipulations, varying the location of special instructions showed that the placement of special instructions within the navigational path increased the likelihood that they were used in answering the question. In addition, larger answer spaces for open-ended questions significantly increased the number of words and themes respondents provided when answering. Further, the use of lines in open-ended answer spaces also influenced respondent answers. Drawing upon the above branching experiments, an arrow directed respondents to a subordinate question and increased the likelihood that the question was answered. The other manipulations involving ordinal scale questions that were also imbedded in the mail survey are discussed later in this thesis. Scalar Questions Scalar questions are one of the most commonly used question formats by all social scientists including sociologists (beginning with the work of Likert in 1932). Different levels of measurement distinguish specific types of scalar questions. Ordinal scale questions require 8 The prevention and detection methods were also used in the Decennial Census with slight modifications based on the previous student experiment (Redline, Dillman, Dajani, and Scaggs, In Press). 10

22 respondents to choose from an ordered set of categories where each category is quantifiably related to each other category. Thus, ordinal scales allow for gradations of meaning to be expressed. In contrast, nominal scale questions require respondents to choose from a list of two or more categories where the relationship among the categories is internal to the scale and cannot be quantified (Singleton and Straits 1999). Ordinal scale questions require respondents to select a category that best represents where they fit along a continuum from positive to negative 9. In constructing ordinal scales for self-administered questionnaires, the visual layout of the scale or continuum may become an important source of information that respondents use when deciding which answer to select. It is also important to distinguish between unipolar scales, scales reflecting varying degrees of some variable, and bipolar scales, scales with a clear midpoint and a positive and negative side. Most attitude scales use bipolar ordinal scales because attitudes are seen as positive, negative, or neutral; and social scientists aim to measure gradations of these attitudes using ordered categories. The verbal language used in question wording is a key source of meaning for both open and closed-ended questions and is an important stimulus in both visual and aural modes. In closedended questions, an equally important source of information is the response categories themselves. Response categories help respondents define their task and give information about the researcher s knowledge of how the answers are distributed in the population (Schwarz and Hippler, 1991). In ordinal scale questions, the meaning of the individual categories is partially determined by how the categories are conveyed to the respondent, either aurally or verbally. Work in survey methodology 9 Interval level of measurement is distinguished from ordinal where there are equal intervals between categories. Singleton and Straits (1999) say, while social researchers often aim to create interval measures, most of what passes for this level of measurement is only a very rough approximation (Singleton and Straits 1999, p. 112). When sequential numbers are used to define sequencing between categories in ordinal scale questions, respondents may then perceive the distance between categories to be equal. 11

23 has focused on the following factors in relation to constructing response scales: the number of scale points, the use of verbal labels and whether to use a midpoint, the use of numeric labels, the use of a "don t know" or "no opinion" filter, and the graphical layout of scales. This summary of research will emphasize the recent contributions of research on visual design in extending our understanding of ordinal scale questions. How many scale points? Considerable work on scalar questions has focused on how many scale points should be used. Krosnick and Fabrigar (1997) argue that longer scales are necessary to represent the entire continuum of possible answers and that longer scales provide increases in the specificity of meaning conveyed to the researcher. However, longer scales reach an optimal point of discrimination where the gains of adding more scale points become quite small and then may add to respondent confusion (McKelvie 1978). Krosnick and Fabrigar (1997) argue that the optimal length for attitude or behavior scales is between 5 and 7 points. Similarly, Bradburn and Sudman (1991) argue that between 5 and 11 points should be used. Thus, when testing the influence of visual design on scalar questions, it seems desirable to use scales somewhere between five and eleven points. Verbal labels. The research on the use of verbal labels is quite diverse. Methodologists argue that fully labeled scales, where all scale points are given a verbal label, are more reliable than polar-point scales where internal points are not labeled (Krosnick and Fabrigar 1997, Schwarz and Hippler 1991). Fully labeled scales may be more reliable when care is taken to choose labels that are relatively precise to help reduce respondent burden. Further, rating scales are less susceptible to context effects when more verbal labels are used (Krosnick and Fabrigar 1991, p. 150). However, Krosnick and Fabrigar (1997) also argue that polar point labeled scales may be less cognitively demanding on the respondent. The research on verbal labels shows that respondents 12

24 gain information from the label given to the category and its position in relation to other categories. Thus, respondents draw on nonverbal information in addition to the verbal labels given to each category. For example, unbalanced scales, where categories were labeled Excellent, Good, Fair, Poor, and Very Poor, produce different responses from those asked using a balanced scale, where the second and middle categories are Very Good and Good (Dillman and Christian 2003). Further, Schwarz, et. al. (1985) find that respondents reported less television watching (mean = 2.7 vs. 3.2 hours) when the scale ranged from less than 1/2 hour to more than 2 1/2 hours than from less than 2 1/2 hours to more than 4 1/2 hours with the middle categories representing half hour increments in both scales. Similar results in a study conducted by Rockwood, Sangster, and Dillman (1997) confirmed these results using the same category ranges. These experiments demonstrate that the verbal labels attached to scale points are an important source of meaning that respondents draw upon when answering survey questions. However, the position of each category or scale point is also an important source of meaning (and may change the meaning that respondents give to the verbal label) that respondents draw upon when answering ordinal scale questions. Midpoint. Whether to provide a midpoint response in scalar questions is also highly debated. In fully labeled scales, the use of a midpoint and a verbal label that defines it as a midpoint, may influence how respondents answer the question. Most agree that when a middle or average answer is offered, more respondents will select that option (Krosnick and Fabrigar 1997, Schwarz and Hippler 1991). Krosnick and Fabrigar (1997) argue that the inclusion of a midpoint may also increase satisficing because more respondents select the middle or average option and fail to optimize by completing all the necessary steps of the question/answer process. The influence of 13

25 the use of a midpoint on reliability is less clear. Thus, the decision to include a midpoint in a response scale should be evaluated for the particular question recognizing that by offering it, it is more likely that it will be chosen. Numeric labels. Evidence also exists that numerical labels influence how respondents answer scalar questions. Numerical labels may be more precise and easier for respondents to hold in their memory (Krosnick and Fabrigar 1997). Numeric labels can be used in two ways: as the main category label (like the number of hours or years) to categorize gradations of behavior or to define the order or sequence of the verbal labels (where 1 means Very Satisfied, 2 means Somewhat Satisfied, etc.) Schwarz, et. al. (1991a) show that changing the numeric labels from 0 to 10 to 5 to 5 (while keeping the verbal labels for the endpoints the same) dramatically influenced how respondents answered the question resulting in less use of the lower categories when they were given a negative label (-5 to -1). For example, respondents rated both success in life and childhood happiness higher for themselves and their parents when given the 5 to 5 scale. Thus, the numeric labels attached to the scale can change the meaning that respondents give to the verbal labels of the scale; and the numerical language can provide an additional source of meaning, which respondents draw upon when answering ordinal scale questions. Don t know filters. Most of the research suggests that the explicit inclusion of a don t know filter results in more respondents selecting don t know." The don t know filter can be included as a full filter, where the don t know option is offered as a separate first question. More common in self-administered surveys is the use of a quasi filter where the don t know option is included along with the other response options (Schwarz and Hippler 1991). The inclusion of either the full or quasi filter often results in more respondents selecting the option, but the endorsement of the full filter is much greater than for the quasi filter. It is necessary to include this option when the 14

26 researcher reasonably believes that a number of respondents will fall into that category because they have no knowledge of or familiarity with the topic addressed; however, if only a few respondents are affected, perhaps a filter can be avoided. Graphical layout. In addition to the graphical location or position of scale labels, manipulating the graphical layout of a scale influences answers to scalar questions. Smith (1993, 1995) showed how different graphical layouts of a scalar question dramatically influenced where respondents placed themselves on a Socioeconomic Status Scale. Surveys in nine different countries asked respondents to subjectively measure where they fit on a social stratification scale. Results for eight countries, excluding the Netherlands, appeared to reflect actual differences in stratification within these countries. However, the results for the Netherlands, which were much lower than those in the other countries, did not coincide with existing information about stratification within Dutch society. Smith (1993) found one difference in the graphical layout: respondents in the Netherlands received a pyramid version where the lower boxes were wider than those on the middle and the top, and respondents in the other countries received a ladder version where equal size boxes were stacked upon one another. Smith (1993) speculated that respondents on the pyramid version were more likely to place themselves lower on the scale, assuming that the researcher s knowledge of the actual distribution meant that the average was closer to the bottom. Schwarz, Grayson, and Knäuper (1998) retested Smith s test (using ladder, pyramid, and onion formats 10 ) on a question asking university students to subjectively rank their academic performance and confirmed these results with lower scores on the pyramid version when compared to the ladder and onion formats. These experiments show that the graphical layout of the response scale can influence substantive answers to scalar questions. 10 The onion format, which was wider in the middle and narrower both at the top and the bottom, produced similar results to the ladder format (Schwarz, Grayson, and Knäuper 1998). 15

27 This research suggests that the categories used in ordinal scale questions are important sources of meaning that respondents draw upon when answering these types of questions. The fully labeled scales may be more reliable and less susceptible to context effects, but polar point scales may be less cognitively demanding on the respondent. Further, respondents determine the meaning of individual categories by the labels used to describe the category, either numerical or verbal, as well as the label s graphical position in relation to other categories in the scale. In addition, the overall graphical layout of the scale can also influence how respondents answer ordinal scale questions. Lastly, providing midpoints and "don t know" quasi filters can increase their use by respondents. The research on understanding how respondents answer ordinal scale questions also indicates that, there is much work left to be done exploring these issues in future research (Krosnick and Fabrigar 1997, p.159). EXPERIMENTAL COMPARISONS Linear vs. Nonlinear Scale Layout Paper Experimental Comparisons and Results. The mail survey discussed earlier included two experimental comparisons of visual language manipulations of a linear and nonlinear version of two scalar questions to extend our knowledge of how visual design influences responses to ordinal scale questions (Christian and Dillman, In Press; Dillman and Christian 2003). Both questions compared a linear layout with response categories presented vertically to a nonlinear layout. One question included categories that were triple banked, and response categories appeared horizontally in three columns. In the second question categories were double banked, and response categories appeared horizontally in two columns. 16

28 In the triple-banked version, where respondents evaluated the quality of education they were receiving, significantly more respondents chose categories from the top line (c 2 = 6.66, p =.010). Most of this difference results from significantly more respondents (c 2 = 8.2, p =.004) choosing the second option Good (40.4% vs. 31.3%) and less choosing the first option Very Good (42.4% vs. 48.8%) when Good appeared to the right versus directly underneath the option, Very Good. In the double-banked version, where respondents evaluated their satisfaction with the variety of classes offered, we found slight differences in the expected direction with slightly more mentions in the top line, but chi-square differences were not significant (Christian and Dillman, In Press). These results indicate that in the nonlinear layouts, some respondents may be reading and processing the categories horizontally and others vertically, while almost all respondents process the categories vertically in the linear format and suggest the need to further test the effects of double- and triple-banked formats. Web Experimental Comparisons. To extend our understanding of the influence of graphical language on responses to scalar questions, Dillman and I developed four different layouts that were implemented using a web survey. We tested unanswered questions from the paper survey using four experimental versions of each question. Since some respondents may read horizontally and others vertically, the horizontal format may be processed differently by respondents, resulting in different answers reported. However, when the options are listed in one linear column, the layout suggests that respondents should process the categories vertically. The horizontal format also interrupts the order of the categories or continuum of the scale that is crucial in conveying ordinal scales. We modeled this experiment after the question on the previous mail survey but used four versions. In this thesis, I compare a linear version to three different nonlinear versions of four ordinal scale questions. 17

29 We repeated the two formats discussed above, a linear format where all responses appeared in one vertical column, and a nonlinear format where responses appeared horizontally across the page and the options read vertically in the web survey (see Figure 1). Since respondents can process the categories vertically or horizontally when they are listed in multiple columns on the nonlinear format, we designed a third version where responses read horizontally. I can compare this version to the nonlinear version where responses read vertically to see whether these visual differences influenced respondent answers to the linear version. The fourth version included numbers on the nonlinear version that read vertically to see whether respondents used the numbers as a guide to show them the order that the response categories should be read, horizontally or vertically. Thus, I can also compare this version to the nonlinear version where responses read vertically to see if the additional numerical language support helps correctly guide respondents through the response categories. Using two questions with five substantive categories (Figure 1, A and B), I can compare the linear version to a triple-banked version where the responses ran horizontally and a triple-banked version where the responses ran vertically, and a version with additional numbers on the version that ran vertically. Using a third question (Figure 1, C), I can compare a scale with four categories and a fifth category "don t know" quasi filter. This question also included the same four versions discussed above: a linear version, a triple-banked version where responses ran horizontally, a triplebanked version where responses ran vertically, and a triple-banked version where responses ran vertically with additional numerical language support. On the final question I can compare similar versions using four categories to see whether the use of only four categories and two columns changed how respondents interpreted the scale. Since the results for the four-category scale on the mail survey were in the expected direction but not significant, it seemed necessary to test a double- 18

30 banked question on the web survey. On this question the versions were similar to the other questions except the nonlinear versions were double banked (see Figure 1, D) resulting in: a linear version, a double-banked version where responses ran horizontally, a double-banked version where responses ran vertically, and a triple banked version where responses ran vertically with additional numerical language support. Fully Labeled vs. Polar Point vs. Answer Box Paper Survey Experimental Comparisons and Results. Another experiment on the paper survey compared a fully labeled scale where all categories were given verbal labels to a polar point scale where only the endpoints were verbally labeled and repeated this comparison using three different questions within the survey. On all three tests of this comparison, respondents were more likely to choose the middle category on the polar point version and the second category on the fully labeled version (Dillman and Christian 2003). Further, the means were significantly higher (showing less satisfaction) on the polar point version in two of the three tests indicating more positive ratings on the fully labeled version (t=7.7, p=.000; t=1.7, p=.045). This experiment showed that by defining the interior categories with verbal labels, respondents answered ordinal scale questions differently. One experimental test included numeric labels but did not directly test the effects of those labels, so this experiment indicated the need to further test the effects of verbal and numeric labels where the effects of adding/removing the labels could be directly compared. Another experiment on the paper survey using ordinal scales compared a polar point format to an answer box where respondents were required to write-in the number corresponding to their answer in an answer box with an instruction, Number of your rating to the right of the answer box. On all three questions, we found significant differences in the mean tests and chi square tests 19

31 with the answer box version producing larger means. Thus, respondents reported more negative responses to questions about their satisfaction with classes, the quality of instruction and advising on the answer box format (Christian and Dillman, In Press). In addition, we conducted further analysis by examining scratch outs and suggest that respondents had difficulty applying the information from the stem of the question at the time of response in the answer box version, as significantly more scratch outs were found (10% vs. 1%); and most of these were because respondents mentally reversed the scale (i.e. changes were from 4 to 2 or 5 to 1, and vice versa). Web Experimental Comparisons. Dillman and I also designed these web survey comparisons to extend our previous knowledge based on the mail experimentation using four different versions of each question. The work discussed above suggests that fully labeled scales are more reliable than polar point ones but that polar point scales may be easier for the respondent to remember when forming an answer; and the mail survey discussed earlier suggests that answers to polar point scales are quite different from those selected on fully labeled scales. The development of telephone interviewing encouraged the use of polar-point-labeled scales rather than fully labeled scales because it is easier for interviewers to ask the question when they only have to say the labeled endpoints. Further, numeric labels as shown by Schwarz et. al. (1991) are also extremely important in influencing how respondents answer scalar questions. Thus, it is necessary to further test the effects of numeric language and compare results with those from fully labeled and polar point scales. This experiment is an effort to further understand how the addition of verbal and numeric labels changes the visual display of the scale and what effects this may have on respondents answers. The answer box format is also closer to the actual stimulus that respondents receive on telephone surveys. In the answer box format, respondents read the question (it is read to them on 20

32 the telephone) where the polar point verbal and numeric labels are provided in the query, and respondents are asked to report their answer using a number. This format does not provide the additional supporting verbal, numeric, symbolic and graphical information that is conveyed to the respondent when the response scale is presented using full or polar point labels. Respondents are then required to apply the information from the query at the time of response with no additional visual language information provided at the time of response. Dillman (2000) speculated that the use of an answer box on self-administered surveys would be equivalent to the stimulus provided in aural modes. Thus, it seems necessary to test the effects of this response format in selfadministered surveys and compare the response distributions with the fully labeled and polar point formats where nonverbal languages can be used to support the verbal languages. I examine four experimental questions using the following four versions: fully labeled version, polar point version with numbers, a polar point version without numbers, and an answer box version (Figure 2, A-D). We designed this experiment to remove additional language support in each version so that both the independent and combined effects of these visual manipulations on respondent behavior could be investigated. We removed the interior verbal labels from the fully labeled version to create the polar points. We also removed the numbers from all scale points on the polar point version. Lastly, we removed the response scale itself, which included verbal, numeric, symbolic, and graphical languages to create the answer box version. Thus, I can compare the polar point version with and without numbers to determine the independent effects of adding numerical language. I can also compare the fully labeled, polar point, and answer box versions to one another. We designed two additional experimental questions to analyze the addition of a quasi filter where we included don t know as one of the response options (Figure 2, E and F). We added the 21