Lecture Scribe February 4, 2016: Questionnaire Design

Transcription

1 Lecture Scribe February 4, 2016: Questionnaire Design Scribe: Julian Brinkley Purpose of Questionnaires Questionnaires are broadly used in human subjects research and are commonly used either to collect demographic information or to collect information about a respondent s opinions or behaviors. Benefits and Drawbacks Questionnaires have a large number of benefits and notable drawbacks. Generally, questionnaires are a relatively easy method of collecting data from a large number of people at a relatively low cost. Although and web page questionnaires are increasingly used, questionnaires generally do not require special tools or equipment and can be done using pencil and paper. As a result questionnaires can be very effective at providing an overview of a population relatively easily and inexpensively (Lazar, Feng and Hochheiser, 101). The use of questionnaires, especially electronic, may also be helpful in minimizing social desirability bias where research participants may be inclined to provide what they view as more socially acceptable answers to researchers. Questionnaire responses are also quantifiable and thus amenable to statistical analysis. There are a few drawbacks to using questionnaires as a research method however. While questionnaires are very good at collecting data from a broad number of people they are typically ineffective at gathering deep, highly detailed data. Given that many questionnaires are self-administered it is often impossible to ask follow-up question or to ask more detailed, probing questions as with an interview. Questionnaires are also subject to bias given that some questions may be poorly worded which may result in multiple interpretations or may lead the respondent. Questionnaire Design When designing a questionnaire it is important to consider the physical format, the type and ordering of questions and the page layout. Physical format: Generally either an electronic or paper format is used to make a questionnaire available to respondents. Both of these approaches have strengths and weaknesses however. A questionnaire in electronic format that is made available online for instance is easy to distribute, is reachable by anyone using a web enabled device, is generally more legible than paper and may ease the data collection and analysis processes. But while an online questionnaire eases access for potential respondents with internet access, this technical requirement may serve as a barrier to those in areas with limited internet access or Cap 5108: Research Methods Lecture February 4,

2 otherwise without the means to access the questionnaire. Online questionnaires hosted with a company like Survey Monkey ( for instance may also entail additional costs for the research team. Paper based questionnaires are generally easier for respondents to complete relative to online questionnaires and responses may be received more quickly. Paper based approaches are limited however in that the responses may be less legible and the distribution area limited when compared to an electronic questionnaire available online. Given that the resulting data is on paper and not in electronic format, collecting and analyzing the data may be more complicated as well. While both approaches have strengths and weaknesses a paper-based approach may be the only option where there is little to no access to computers. An electronic approach may be necessary when dealing with populations that are visually impaired or have a reading impediment or that otherwise may not be able to use a paper questionnaire. Type of questions: A key consideration in questionnaire design is the development of well-written, unbiased questions. Poorly written question may lead to respondent confusion and elicit answers that are not reflective of what was being asked. Well-worded questionnaires consist of questions that are easily understood and unambiguous. To this end survey question are often either open ended or closed ended and employ rankings or ratings. Open-ended questions, or questions without a prescribed response, provide respondents with an opportunity to answer a question in their own words. These types of questions, relative to closed ended questions, may help researchers in gaining a better understanding of a topic by giving respondents complete freedom in their answers. An example of an open-ended question about a respondent s use of the Chrome web browser could be worded as: How do you feel about the usability of the Chrome web browser? Open-ended questions are not without drawbacks however. Open-ended questions can be difficult to word in such a way as to be simple while remaining unambiguous. Openended questions also require more complex data analysis. Closed-ended questions are questions with a limited number of responses. Closed-ended questions are generally easy to answer and factually based. These types of questions are also more amenable to data collection and analysis. An example of a closed ended question about the Chrome web browser could be worded as follows: Do you like using the chrome web browser? Closed-ended questions like open-ended questions have drawbacks. Closed-ended questions generally restrict the respondent s answers and as a result have limited effectiveness in terms of eliciting deep feedback. Cap 5108: Research Methods Lecture February 4,

3 Ranking questions are closed ended questions that ask a respondent to create an ordered list from provided options whereas ratings ask respondents to respond on a scale. A ranking question regarding web browsers could ask a respondent to rank in order, from favorite to least favorite, the following web browsers: Google Chrome, Internet Explorer, Mozilla Firefox and Safari. Unlike a ranking question a rating question could ask a respondent to indicate their feelings about the Google Chrome web browser on a scale from strongly dislike to strongly like. Rating scale questions are either unipolar or bipolar. A unipolar scale asks a respondent to identify the presence or absence of an attribute (e.g. satisfaction ) and often uses a five point Likert scale (Lazar, Feng, Hochheiser, 112). A bipolar scale asks a respondent to choose within a range of polar opposites (e.g. satisfaction versus dissatisfaction) with the presence of a midpoint ( neither satisfied or dissatisfied). Ranking questions are commonly viewed as being more reliable than rating questions as rating questions may prompt a respondent to pick a relatively neutral median rating. Ordering of Questions: Question ordering is a key factor in questionnaire design and one where inattention to detail may limit the questionnaire s response rate. Asking very personal questions about age, education or personal income up front may discourage respondents from completing the questionnaire. It is generally recommended that these personal demographic questions be asked at or towards the end. Lazar, Feng and Hochheiser have recommended asking potentially interesting questions early on with the hope of piquing the interest of a respondent and encouraging them, through the questions being asked, to continue the questionnaire through its completion (114). It is also recommended that related topics be grouped together and that questions that respondents might be reluctant to answer should be asked multiple times in perhaps slightly different ways. Page layout: The layout of the questionnaire should be given careful consideration prior to the questionnaire being released. Instructions as applicable should be clear and easy to understand. A respondent should understand clearly if they should circle their responses, mark them with an X or check in a box for instance. If a questionnaire is broken down into sections those section should be clearly delineated from one another. Delineation plays a large part in the overall readability of the questionnaire and may also be supported through the judicious use of white space and colored bars to separate questions (Figure 3). A poor layout or a poor use of whitespace may lead to users accidently selecting an unintended response. Cap 5108: Research Methods Lecture February 4,

4 Figure 3: An implementation of the System Usability Scale questionnaire that uses alternating gray bars and evenly spaced checkboxes to improve readability. Cap 5108: Research Methods Lecture February 4,

5 Questionnaire Testing It is very important to pre-test a questionnaire prior to distributing it to potential respondents. Pre-testing may help to ensure that questions are easy to understand and unambiguous. Some questions may also be written in a way that prompts everyone either to agree or disagree en masse. Pre-testing allows for early analysis and the use of techniques, like means analysis, to determine if questions are written in ways that are not discriminating. Pre-testing also helps to ensure that the questionnaire is physically readable in the case of a paper based approach or accessible online in electronic approaches. In the case of paper-based questionnaires the specific font face used and line spacing may impact respondents perceptions of ease of use. Similarly, user interface issues pose a problem for a hosted questionnaire or one distributed using . Pre Existing Questionnaires While not specifically discussed during the lecture there are a large number of questionnaires that exist, in the human computer interaction domain especially, that have been widely used and broadly discussed in the literature. Two common questionnaires are the System Usability Scale (SUS) questionnaire (Brooke, 1996) and the Questionnaire for User Interface Satisfaction (QUIS) (Chin, Diehl and Norman, 1988). The SUS is a 10- item, five point Likert scale (from 1 = strongly disagree to 5 = strongly agree), that provides a global view of system usability through a weighted score in the range of (Brooke, 1996; Bangor, Kortum & Miller 2009). The QUIS measures a respondent s overall satisfaction with a system s user interface across six scales involving nine specific interface factors (screen factors, terminology, system feedback, learning factors, system capabilities, technical manuals, online tutorials, multimedia, teleconferencing and software installation). Each of the aforementioned factors is measured on a 9-point scale. The key benefit to using either of the aforementioned tools, or using an existing tool generally, is that it potentially eliminates the need to develop one from scratch and presumably has broad support within the academic literature. Cap 5108: Research Methods Lecture February 4,

6 Works Cited Bangor, Aaron, Philip Kortum, and James Miller. "Determining what individual SUS scores mean: Adding an adjective rating scale." Journal of usability studies 4.3 (2009): Brooke, John. "SUS-A quick and dirty usability scale." Usability evaluation in industry (1996): 4-7. Chin, John P., Virginia A. Diehl, and Kent L. Norman. "Development of an instrument measuring user satisfaction of the human-computer interface."proceedings of the SIGCHI conference on Human factors in computing systems. ACM, Lazar, Jonathan, Jinjuan Feng and Harry Hochheiser. Research methods in humancomputer interaction. Chichester: John Wiley & Sons, Print. Cap 5108: Research Methods Lecture February 4,