COMP 388/441 HCI: 04 - Evaluating Interface Designs Evaluating Interface Designs

Transcription

1 04 - Evaluating Interface Designs

2 Lecture 04 - Overview Introduction & Determinants Do not rely on your intuition when it comes to evaluating usability Overcome intuitive judgment by arranging extensive testing and evaluation procedures Note: Do not confuse Usability Testing with traditional software testing A variety of strategies/methods are available to support Usability Testing ; which one(s) to chose and how to configure them individually depends on a set of determinants: Stage of design (e.g., early, late, etc.) Novelty of the project/product (well defined vs. exploratory) Number of expected users Criticality of the interface (e.g., life-critical vs. home office) Cost of product & finances allocated to testing Available time Experience of the design and evaluation team

3 Lecture 04 - Overview Motivation & Challenges Motivation COMP 388/441 HCI: 04 - Evaluating Interface Designs Interest in usability has grown rapidly in the past years Competition is higher Possible monetary or legal consequences in cases of failure or non-compliance with contracts Challenges Uncertainty of testing: You can never assume that the UI is error free, hence you need to plan for continues improvements throughout the entire product life-cycle; expect change requests from clients Deadlines: At some point you must release the product even if finding additional errors is anticipated Unpredictable situations: You need to think beyond normal usage scenarios and also consider exceptional/unpredictable situations (e.g., test under critical conditions)

4 Lecture 04 - Overview Testing & Evaluation Strategies/Methods 1. Expert Reviews 2. Usability Testing and Laboratories 3. Survey Instruments 4. Acceptance Test 5. Evaluation During Active Use Typical Use Validation of Requirements 4 Identification of Flaws 1, 2, 3 5 Before Release After Release

5 Expert Reviews

6 Expert Reviews Overview COMP 388/441 HCI: 04 - Evaluating Interface Designs Experts (experienced in application domain and/or UI domain), commonly consultants or staff, perform a review of the UI General issues: When: Early or late, in general during several points of the development process (time and number depending on progress, availability of experts, design team ready?, budget) Who: The same or new experts for every review (tradeoff) Duration: From a couple of hours to weeks Outcome: (A) Formal report including identified problems and recommendations for changes and/or (B) Presentation and discussion with the design team

7 Expert Reviews Expert Review Methods Heuristic Evaluation: Review UI to determine compliance with a short list of of design heuristics (e.g., The 8 golden rules of UI design ) Guidelines Review: Review UI for conformance with the guidelines document Consistency Inspection: Verify consistency across several UIs, within a UI, or within a tutorial Cognitive Walkthrough: Simulate user, use typical tasks (e.g., high frequency task, critical tasks, error handling) Formal Usability Inspection: Experts participate in a meeting/discussion with a moderator who presents the interface and asks specific questions

8 Expert Reviews Review Approaches/Reporting Techniques After choosing a review method you need to select the right approach/means and/or reporting style to implement it. Some examples: Ranked Recommendation (assign priorities) Birds-Eye View (study printed screens from distance) Use of Software Tools (speed up the review process) General challenge: Experts may lack an understanding of the task domain and/or user community, or may be biased. Hence it is crucial to chose knowledgeable experts that are familiar with the project and organization.

9 Usability Testing and Laboratories

10 Usability Testing and Laboratories Overview Actual potential users test the UI, commonly in a lab-environment Benefits: Provide supportive confirmation of the progress and recommendations for changes, speeds up the development process and reduces costs 2 general strategies: Controlled Experiment (traditional) Purpose: Test a hypothesis, support theories, seek to show statistically significant differences in 2+ issues Number of participants: Many Outcome: Validation or rejection of a hypothesis Usability Tests (UI focused) Purpose: Refine interfaces Number of participants: Few Outcome: Report with recommended changes

11 Usability Testing and Laboratories Usability Laboratories Host the tests, have typically (not always) two areas separated by a one-way mirror The user is observed by a usability tester or designer through the mirror Are commonly staffed with 1+ experts in UI design

12 Usability Testing and Laboratories Usability Test Procedure: Pre-Test Develop a test plan Test Plan contains a list of tasks, subjective satisfaction and debriefing questions, identifies number, types and sources of participants Developed in collaboration with the design team, also consider deadlines and budgets Conduct pilot test Usually a couple of weeks before the actual test Very limited number of participants Purpose: Validate procedure, tasks, questions, adjust if problems occur Choose participants Selection Criteria: background, experiences with the tasks, motivation, education, natural language ability, physical abilities Also consider issues such as time, date, noise, etc.

13 Usability Testing and Laboratories Usability Test Procedure: During the Test Techniques (examples) Ask users to think aloud about what they are doing, possibly use two participants that talk to each other; as the tester: be supportive, listen for clues Videotape participants; this enables later review and demonstration to designers/developers; logging during the test is necessary! After the task list is accomplished, optionally ask participants to make general comments and suggestions or ask specific questions The result of the test can be used to improve the UI; the improved version builds the basis for the next test (iterative approach - cf. LUCID)

14 Usability Testing and Laboratories Usability Test Procedure: Dealing with users Always treat participants with respect Let them know that it is not them being tested but the UI Participants need to be told what they are expected to do and how long the test will take (including time before and after the actual test) In order to give the test a more professional touch and/or to avoid legal consequences it is typical to let participants sign a statement that includes, for instance, issues such as the right to withdraw or the voluntary nature of the participation, etc.

15 Usability Testing and Laboratories Different Forms of Usability Testing Paper Mockups Discount Usability Testing Competitive Usability Testing Universal Usability Testing Field Tests and Portable Labs Remote Usability Testing Can you break this Test

16 Usability Testing and Laboratories General Limitations Time-based restrictions Emphasis on first-time usage Limited coverage Location-based Lab-Environment unrealistic

17 Survey Instruments

18 Survey Instruments Overview Supplement to expert reviews and usability tests Benefits over expert reviews and usability tests: less bias, the large number of participants tends to make the results more significant, inexpensive In order to have success the development of questions as well as the analysis of the collected data needs to be performed carefully Prepare, test and review the questionnaires with a small number of participants before conducting the large scale survey Develop the analysis and presentation before the survey distribution; this is more focused as opposed to unplanned statistical gathering You need to build-in methods to verify your target group (e.g., age, gender, experience, motivation, etc.)

19 Survey Instruments Kind of questions and their goals 1. General questions, example: Response to a statement on a scale (e.g., The UI enables me to do X and Y more efficiently, scale: strongly agree, agree, neutral, disagree, strongly disagree) Identifies problems that users have with the UI, uncovers where improvements are necessary and helps to demonstrate how UIs improve over time 2. Detailed questions, example: Evaluate certain aspects of the UI (e.g., purpose of commands, helpfulness of error messages) Are more precise, feedback commonly more useful in regard to taking specific actions

20 Survey Instruments Reference Questionnaires COMP 388/441 HCI: 04 - Evaluating Interface Designs QUIS (Questionnaire for User Interaction Satisfaction): Covers UI details such as readability of characters, terminology, purpose of icons, etc. IBM s Post-Study System Usability Questionnaire: Focuses on system usefulness, information quality, interface quality Software Usability Measurement Inventory: Is designed to measure users perceptions of their effect, efficiency, and control WAMMI s Web Usability Questionnaire: Web-based evaluations

21 Acceptance Tests

22 Acceptance Tests Is used to confirm a prior defined set of (measurable), criteria, objectives or goals Why?: It is crucial that the product fulfills those criteria in order to comply, for instance, with a contract Important: The goal here is NOT to identify flaws (cf. expert review, usability tests, surveys) but to verify adherence to requirements ToDo: Develop explicit requirements when you write the requirements document or when a contract is offered to the client Note: The criteria can also come directly from the client Example for such criteria are the usability measures from chapter 1 It is crucial to develop precise criteria, userfriendliness is of less relevance, the goal is rather to validate the proper defined criteria In order to guarantee neutrality of the results, acceptance tests are often performed by outside testing organizations

23 Evaluation During Active Use

24 Evaluation During Active Use Overview After the release of the product, constant improvement, refinement is necessary Demonstrating that you are considering feedback and that improvements will be made in a future release will be appreciated by the user 5 strategies/means: Interviews and focus-group discussions Continuous user-performance data logging Online or telephone consultants Online suggestion box or trouble reporting Discussion groups and newsgroups

25 Evaluation During Active Use Interviews and focus-group discussions Pros: Can be productive, the interviewer can focus on specific issues of concern, confirmation of one interview with a series of interviews possible, specific constructive suggestions can be obtained, in professionally led focus group interviews it is possible to uncover hidden problems or usage pattern, which can then quickly be explored and confirmed by the participants Cons: Costly, time consuming, only consider a small fraction of the user community (significance?)

26 Evaluation During Active Use Continuous user-performance data logging Automated logging of user interactions with the UI (e.g., patterns of usage, speed of performance, rate of errors, frequency of onlinehelp request) Provide guidance for the acquisition of new hardware, changes in operating procedures, improvement of training Examples: Error frequencies: Change messages, revise training material, simplify command syntax Use frequencies: Simplify access to certain features This only works if there is a logging mechanism in place, otherwise you will not have any data Always consider user privacy (e.g. avoid logging usernames)

27 Evaluation During Active Use Online or telephone consultants Main purpose: Provide helpful and effective personal assistance However: They are also an excellent source of information about problems that users have and can suggest improvements and potential extension Realized via phone (possibly even with remote desktop access) or life (real-time) chat Offering such services builds customer loyalty and provides insights that can lead to better UIs

28 Evaluation During Active Use Online suggestion box or trouble reporting online error reporting (e.g., via or websites, automatic reporting tools, etc.) May encourage users to make productive comments Easier than writing a letter or calling

29 Evaluation During Active Use Discussion groups and newsgroups Suitable if there is no individual available / in the user s mind that can help with trouble shooting Usually the manufacturer of a product provides discussion forums or newsgroups Usually provide a list of topics, users can scan topics and add new topics Commonly involve a moderator

30 The End Questions?