Usability Test Results CyberLynx Annotator for Lotus 1-2-3 Executive Summary John Milanski ENG525 Usability Testing 25 Mar 1997 1
2
O VERVIEW Formal usability tests how well your software performs with real users in the real world. A typical user performs set tasks under close scrutiny. By studying what the user does and says, we can determine how easy the product is to use and how to improve it. M ETHODOLOGY user selection It is very important to select testers who are likely users of the product. This seems obvious but is often overlooked. Very strict criteria are established. In this case, we selected an Administrative Assistant at a large accounting firm in Chicago, The user had experience with electronic spreadsheets (Microsoft Excel) and had created annotations in the past. Some instruction in Lotus 1-2-3 was required to overcome differences. test scenario To set the stage for the test, the user was given the role of a junior accountant. The user was asked to perform the following written list of tasks: open an existing spreadsheet file annotate a single cell in that file using CyberLynx Annotator annotate a range of cells in the same file edit and rename an existing annotation repeat these tasks on a second file save the files and exit Lotus 1-2-3 think-aloud protocol People frequently say one thing and then do another. That s what makes data from surveys so unreliable. Intentionally or not, the way questions are written, the tone of the survey taker influence the answers given. Conducting a truly unbiased survey is very hard indeed. That is what makes watching a person s behavior with a new product so useful. The responses are less biased. But they are also ambiguous. The think-aloud method complements behavioral observation by getting into the user s head. In this method, users are asked to simply say what they are thinking as they use the product. With this information, analysts can determine whether a frown was a problem with the user s screen or just an itch on their nose. In this test, think-aloud mainly confirmed results. It keeps telling me to annotate in the yellow box, but there s no box! Maybe I m missing a step. I m lost. Ah! There s those tool tips I was looking for... testing and data collection Testing was performed at the Usability Management test facility in Skokie, IL, on Feb 25, 1997. The user was placed in an office-like room and observed through a one-way mirror. Audio and video records of the test were made as well as real-time notes using a computer logging device. Three cameras were placed in the room to record the experience: one focused on the computer screen, one on the nearby instruction manual, and one directly in front of the user. Although left alone in the room, the user could call for technical assistance as a last resort. 3
Before the test began, the user was given basic instructions, They were to perform several annotations on an existing electronic spreadsheet using Lotus 1-2-3 and CyberLynx Annotator. A written script outlined the steps and tasks to be performed. The user was encouraged to think-aloud as she performed the tasks to help the analysts understand her actions. The user was also encouraged to consult the written documentation before asking for help with a problem.. After the test was completed, the user completed a post-test questionnaire with the help of one of the analysts. In all, the test lasted approximately 1 hour 20 minutes. analysis Analysis is a winnowing process. Thousands of pieces of data are collected during the test. Clients who witnessed the test sit down afterward and begin looking at the pieces, one by one, trying to group them in categories. The categories we used represent common problems in software development: concept: visual cues: interaction model: navigation: feedback: terminology: consistency: content: user assistance: Does the user understand the idea of a spreadsheet and of attaching notes to it? Did the user make a mistake because the graphic didn t make sense? Is the user confusing interface concepts (yellow is the Annotator text box)? Does the user get lost moving around the application? How does the user know they ve accomplished the task? Were the terms used on screen/in the documentation confusing, ambiguous? Does the system react to the same action differently in different places? Are the data or pictures in the application confusing the user? Is the documentation bad? Is there a lack of help? project context: Was the roble related to the testing methods or environment? Grouping the observations in this way allows us to quickly see the big problems the user had with the software. Equally important, it leads to better solutions, solutions that solve multiple problems without creating new ones. See Chart 1. Finding # Log # Category Description 1 2 INT trouble opening Annotator 2 2 HELP documentation didn't help user open Annotator 3 2 CUE screen cues did not help her locate the product and open it 4 3 PROJ opening Lotus 1-2-3 was not on the task sheet 5 7 HELP spending a lot of time studying task list 6 6 9 FEED user selects a range of cells but the annotation doesn't change 7 8 9 10 Category CONC CUE INT NAV FEED TERM CONS CONT HELP PROJ Abbreviations concept visual cues interaction model navigation feedback terminology consistency content user assistance project context Chart 1. Sorting data based on categories (example). 4
R ESULTS: MAJOR USABILITY PROBLEMS 1. User had trouble starting the Annotator the first time The decision was made during the product development process NOT to automatically load the Annotator and place its icon on the toolbar when Lotus 1-2-3 opened. The users must place the icon on the toolbar on their own. 2. The test script was not detailed enough. In places confusion was caused simply because a step had been left out of the written script. A pilot test using the script would solve many of these problems. 3. The user was unable to annotate a range of cells correctly. The user was confused because when she highlighted a cell, the annotation from the previous cell was still in the Annotator window. The user was unsure whether to type over it. This is a failure in feedback. 4. The user was unsure how to save the annotations. The documentation says that the only way to save the annotations is to save the file. This gives the impression that if they type an annotation and click out of the cell without saving the file, the data will be lost. 5. The user wanted several features not available in the product. Since the user usually works in Microsoft Excel, she would like the Annotator available for that program. The user might consider buying the Annotator is they knew how the annotations are printed and how much the software cost. 6. The user became more proficient with the product with repetition over time. Chart 2 illustrates this experience-based improvement. Task Begin End Elapsed Time Comments 1 Open file 3:36 15:20 11:44 trouble opening Annotator from within Lotus 1-2-3 2 Annotate one cell 15:43 26:39 10:56 difficulty saving annotation 3 Annotate a cell range 26:39 29:13 2:34 annotates wrong cell range (poor instructions?) 4 Edit old annotation 29:13 32:19 3:06 --- 5 Repeat task 2&3 32:19 34:49 2:30 --- 6 Save file/exit Lotus 34:49 35:40 0:51 --- Chart 2. Time required to perform tasks. 5
C ONCLUSIONS CyberLynx Annotator has several major usability problems, but they are solvable. Problems opening the Annotator can easily be solved by automatically placing an icon in the Lotus toolbar. Problems annotating and saving the annotations can be fixed by providing more feedback. These problems arose not so much from poor choices by the development team but from delayed user testing of the product. If true user testing is done early and frequently during the development process, mistakes are caught sooner, are easier to fix, and are less costly. For example, the Annotator icon should automatically be placed on the Lotus 1-2-3 tool bar because people who buy the tool intend to use it. Of course the major problem is one of market research. Due to limited resources, the development team had to choose one platform to build toward. It should have been Microsoft Excel, who now has 90% of the market and not Lotus 1-2-3 which is under 5%. 6
A PPENDICES testing scenario and tasks video clips testing log post-test questionnaire and results 7
V IDEO CLIPS Task #1 Testing room setup at usability lab. Task #1 User having trouble getting started. Cannot open Lotus and CyberLynx Annotator together. Task #1 User has now opened Lotus but does not know how to open CyberLynx Annotator. Task #2 User annotates a single cell. Task #2 Unsure how to save the annotation, the user is hopelessly searching the toolbar.
Task #2 Between hunt-and-peck sessions, a frustrated user looks in the documentation to save the annotation. Task #2 After 10 minutes, user finally manages to save the annotation and the document. Task #3 User annotates the wrong cell range. Task #6 User saves the file and exits Lotus 1-2-3.
P OST-TEST TEST QUESTIONNAIRE RESULTS Question Difficulty Comments 1 2 (easy) Only trouble was figuring out how to annotate... 2 1 (very easy) Once I got (the Annotator) up there, clicking on it and saving was very simple. 3 1 (very easy) 4 2 (easy) 5 2-3 (almost easy) The important things were highlighted...but I had to jump around more than I would have liked...it was harder to find things than I would have liked. 6 Yes But I would expect an icon (for the Annotator) on the toolbar. Using yellow (for the Annotator box) is nice. 7 Yes I often need to make notes in a spreadsheet. Now I usually do it in other cells. One downfall is the text is not right next to the cell. I d be interested in seeing how it prints the notes out. 8 Getting started. It s not friendly at first. 9 Clicking outside the cell saves (the annotation) automatically. 10 Yes But before making a buying decision, I d need to see how it prints the notes, if I could get it for Excel, and I d have to play with the program some more.