Chapter 14: Usability testing and field studies The aims: Explain how to do usability testing through examples. Outline the basics of experimental design. Discuss the methods used in usability testing. Discuss the role of field studies in evaluation. Usability testing Basic elements: A set of typical tasks. Controlled environmental settings. Users. Data recording equipments. The data performance times errors. User satisfaction. Experimental study controlled evaluation of specific aspects of interactive behaviour evaluator chooses hypothesis to be tested a number of experimental conditions are considered which differ only in the value of some controlled variable. changes in behavioural measure are attributed to different conditions Example There are many different input devices: keyboard, button, knob, mouse, touch screen, voice activation, etc. Questions: does the optimality of an input device depend on the task being performed? Is performance with a given input device influenced by the age of the user? Example Three factors: Input devices Task demands User capacities How can we categorize these factors? 1
Categorize input devices Direct devices: no translation is required between the activity performed by the person and the action of the device Touch screen Light pen Voice activation Indirect devices: require a translation between the activity of the person and the action of the device Mouse Trackballs Joysticks Rotary encoders Devices Direct Indirect Advantages Direct hand-eye coordination No memory loading Minimal training. Can adjust controldisplay ratio More precise Given tactile feedback disadvantages Arm fatigue Limited resolution Slow entry Finger or arm may obscure screen Requires translation between: rotary and linear movement; hand and screen, Learning time Control time. Hypothesis prediction of outcome framed in terms of IV and D E.g. the optimality of an input device is depending on the task being performed null hypothesis: states no difference between conditions aim is to disprove this E.g. the optimality of an input device is independend from the task being performed Variables independent variable (IV) characteristic changed to produce different conditions e.g. interface style, number of menu items dependent variable (DV) characteristics measured in the experiment e.g. time taken, number of errors. Experimental factors Subjects who representative, sufficient sample Variables things to modify and measure Hypothesis what you d like to show Experimental design how you are going to do it Experimental design within groups design each subject performs experiment under each condition. transfer of learning possible less costly and less likely to suffer from user variation. between groups design each subject performs under only one condition no transfer of learning more users required variation can bias results. 2
Analysis of data Before you start to do any statistics: look at data save original data Choice of statistical technique depends on type of data information required Type of data discrete - finite number of values continuous - any value Analysis - types of test parametric assume normal distribution robust powerful non-parametric do not assume normal distribution less powerful more reliable contingency table classify data by discrete attributes count number of data items in each group Analysis of data (cont.) What information is required? is there a difference? how big is the difference? how accurate is the estimate? Experiments & usability testing Experiments is to discover new knowledge Usability testing is to check that the system is usable by the intended user population for their tasks. Experiments may also be done in usability testing. Usability testing & research Usability testing Improve products Few participants Results inform design Usually not completely replicable Conditions controlled as much as possible Procedure planned Results reported to developers Experiments for research Discover knowledge Many participants Results validated statistically Must be replicable Strongly controlled conditions Experimental design Scientific reported to scientific community Usability lab with observers watching a user & assistant 3
Portable equipment for use in the field Testing conditions Usability lab or other controlled space. Emphasis on: selecting representative users; developing representative tasks. Tasks usually last no more than 30 minutes. The test conditions should be the same for every participant. Informed consent form explains procedures and deals with ethical issues. Some type of data Time: to complete a task. time away from the product. Error Number and type of errors per task. Number of errors per unit of time. Number of navigations to online help or manuals. Number of users making a particular error. Number of users completing task successfully. Usability engineering orientation Aim is improvement with each version. Current level of performance. Minimum acceptable level of performance. Target level of performance. How many participants is enough for user testing? Depends on: schedule for testing; availability of participants; cost of running tests. Typically 5-10 participants. Some experts argue that testing should continue until no new insights are gained. Field studies The aim is to understand what users do naturally and how technology impacts them. Field studies can be used in product design to: - identify opportunities for new technology; - determine design requirements; - decide how best to introduce new technology; - evaluate technology in use. 4
Data collection & analysis Observation & interviews Notes, pictures, recordings Video Logging Analyzes Categorized Categories can be provided by theory Grounded theory Activity theory Key points Testing is a central part of usability testing. Usability testing is done in controlled conditions. Usability testing is an adapted form of experimentation. Experiments aim to test hypotheses by manipulating certain variables while keeping others constant. The experimenter controls the independent variable(s) but not the dependent variable(s). There are three types of experimental design: differentparticipants, same- participants, & matched participants. Field studies are done in natural environments. Typically observation and interviews are used to collect field studies data. Categorization and theory-based techniques are used to analyze the data. 5