Experimental comparison of two help systems for Microsoft Excel according to usability criteria

Transcription

1 Experimental comparison of two help systems for Microsoft Excel according to usability criteria Anke Weidlich Fraunhofer-Institute for Telecommunications Heinrich-Hertz-Institut (HHI), Einsteinufer 37, Berlin, Germany Abstract This contribution describes an experimental investigation comparing two Help systems for Microsoft Excel, firstly Microsoft's built-in Help system, and secondly an online Help system newly developed by atsolute. Although both Help systems structure the help topics hierarchically, they differ in the manner in which the information is presented. The experimental comparison described here analyses which type of representation used in the help systems best meets the criteria of usability. Key words: Excel, Help systems, hierarchically structured information, usability, usabilitytesting 1. Introduction Microsoft's Excel program is extremely widely known and is used by very many users for a wide variety of purposes. Yet a sizeable proportion of users do frequently find themselves faced with diverse user problems when working with Excel. Microsoft has provided a Help system to deal with such eventualities. A significant number of users cannot cope with the Help system, however, and as a result either fail to find the information they require or embark on detours, thus taking more steps and more time for their search than is really necessary and in the end often leading them to seek support from others. Two methods of dealing with such usability deficits present themselves. One obvious way is to try and avoid the ergonomic problems arising from working with Excel an approach which may make sense, but which is not generally when using applications as extensive and wide ranging as Excel. The other obvious method is to improve the Help system itself. This latter path was the one chosen by atsolute, a Berlin firm which in the course of process optimization and software support has specialized in the visual representation of complex knowledge. The remodeling was, however, only performed for one of the four Excel Help systems which Microsoft provides, namely the Contents tab system. The other three MS Help systems were not similarly treated (see Figure 1 below). As with Microsoft's Contents tab Help system atsolute's access to help topics is hierarchically structured. Three features distinguish atsolute's approach from that of Microsoft: 1. a smaller number of options and choices facing the user at each level of the hierarchical structure, from which it follows that the atsolute system requires a larger number of navigational steps than does the Microsoft system; 2. different, and in particular shorter labeling of the options and choices users can select (with the additional possibility of obtaining further information in the form of Bubble Help);

2 3. the use of graphic elements to depict the relationships between menu options on higher and subordinate levels of the hierarchy (the navigation path already completed is shown highlighted in color). In order to explain these differences, a description of each Help system is given in the next chapter. The aim of the investigation was to obtain an answer to the question of whether the manner of remodeling exhibited by atsolute's new Help system stands up to usability criteria when regarded in direct comparison to Microsoft's Help system. This was examined experimentally by means of user testing. 2. Two Help Systems for Microsoft Excel This paper focuses on two alternative Help systems for Microsoft Excel, each offering solutions to a wide variety of problems faced by users. Microsoft's Help topics are usually accessed directly from the Excel program itself, they can however also be obtained online by connecting to the Microsoft website. The Help system provided by atsolute on the other hand is generally presented as a web version, although it is possible to install it locally as well Microsoft's Help System for Microsoft Excel The built-in Microsoft help feature offers users four different ways of obtaining assistance: the animated Office Assistant, the Contents tab, the Index tab and the Answer Wizard tab. Only the hierarchically structured help topics of the Contents tab (see Figure 1 on the right hand side) will be considered here, as this was the only Help feature to be tested in the investigation described below. In order to make use of this feature users must go to the "Help" section of the main menu and open Microsoft Excel Help, one then reaches the Contents tab by clicking on "Content". The left pane of the Contents tab window serves only to allow navigation through the hierarchically structured Help topics. The topics are arranged hierarchically into on four levels (see Figure 1 on the right hand side). At the top level, information is grouped into 35 chapters. Proceeding from this top level, the user has to make a number of choices before reaching the information sought-after; this is then displayed in the right hand pane. Figure 1: The Microsoft Excel help topics (German version)

3 2.2. atsolute's Help System for Microsoft Excel As with the Contents tab window of Microsoft's Help system, the topics provided by atsolute's Help system are arranged in a hierarchical structure. Unlike Microsoft, however, atsolute has chosen a graphic representation of the choices available between alternatives on the various levels (see Figure 2 below). The boxes are labeled and linked as in a hypertext. Figure 2: The "knowledge-management-tree" in atsolute's Excel Help system On beginning navigation, users are shown just one "Start" graphic, upon activation of which the next level appears, showing five labeled sub-level graphics. These sub-level graphics in turn offer new choices, allowing users to refine their search until they find the necessary information. If in the course of navigation one continues to click open the graphics, the result is a wide, branching structure, which atsolute refers to as a "knowledge management tree". The graphics not only serve as the means with which to navigate, they also illustrate the steps to be taken (see Figure 3 on the right hand side): once the user has reached the information he has been looking for, he can see a number of vertically linked graphics, depicting the path to the solution. In order to solve his problem he has to follow the instructions given by the graphics starting at the top and working down. If the user has problems either with a particular graphic or in following the instructions given and thus requires further information, he can click on the graphic concerned and in this way obtain a further series of vertically linked graphics giving detailed information concerning the step he has been unclear about. 3. Test description and test set-up Figure 3: Information to solve a problem in atsolute's Excel Help system 3.1. Tasks Given the extensive range of applications and the variety of tasks which can be tackled using Excel, it was necessary to select tasks appropriate to the purposes of the investigation. The

4 choice of tasks was intended to be representative of typical Excel use, and may be classified into nine categories: 1. Excel settings 2. Data management 3. Database management 4. Analysis of data 5. Spreadsheets 6. Formulae and functions 7. Diagrams, charts and maps 8. Layout 9. Others The choice of tasks emphasized the categories: spreadsheets; formulae and functions; and diagrams, charts and maps. The test subjects were asked to perform the following tasks: 1. Alter the manner in which numbers are displayed in the Excel spreadsheet provided so that zero values are no longer visible. 2. Enter a formula which selects the highest number from a given set of numbers. 3. Alter the scale of the y-axis in one of the two diagrams shown so that it becomes equivalent to the scale shown in the other diagram. 4. Enter the tabulated information contained in a text file in such a way that the tabular information (ordering into rows and columns) is preserved. 5. Add page numbers to the diagram you have created, without typing the numbers in. 6. Modify the Excel spreadsheet provided so that one can enter only whole numbers larger than Investigative method The investigation was designed to ensure that each test subject was familiarized with both Help systems, so as to be in a position to compare and evaluate them subsequently. To do this each test subject performed a total of six tasks, each of which was carried out using first the atsolute Help system and then following on using Microsoft's Help system. Each person was tested for a total of two hours. Searches were regarded as having been ly performed, if the test subjects found information pertaining to the solution to a problem within ten minutes. Where they failed to do this, the test director stopped the search after ten minutes. Once the test subjects had either found the information needed to solve the task or the search had been halted they were asked to implement the solution described in the Help system. Where the search had been stopped the test subjects were shown the solution by the test director. If after five minutes the test subjects had not succeeded in solving the task, implementation of the solution was stopped by the test director. The efficiency of searching and implementation were determined by the time and the number of steps users took access the help information and subsequently put it into effect. In addition, data was collected on user satisfaction with regard to searching for and implementing help information. After completing a task each test subject was asked to answer questions concerning the following aspects: - Orientation and certainty as to which steps to take - Enjoyment, nervousness and boredom experienced - Acceptability of the time taken to perform the task - Overall acceptability Test subjects were required to have experience in working with computers and the internet. In addition they had to be proficient in written and spoken German, this being necessary both to understand the tasks and in order to perform them. 31 test subjects took part in the investigation. They were drawn from all age groups over 15 years, both sexes and all levels of education.

5 4. Results The mean of the results concerning effectiveness and efficiency in accessing and implementing the Help information was calculated separately for each task and Help system. Usability criteria for which the difference between the groups is less than significant are distinguished by asterisks (* for a level of 5% or ** for a level of 1%). The test subjects succeeded in finding the information required to perform the task in 60% of the tasks performed using Microsoft's Help system and in 43% of the tasks performed using atsolute's Help system. The difference is significant. When using Microsoft's Help system, test subjects were able to implement the solutions obtained in 92% of cases compared with 95% when using atsolute's Help system. This difference is not significant. Successfully performed searches over all tasks in per cent search solving * Microsoft atsolute 0% 20% 40% 60% 80% 100% Figure 4: Help information ly accessed and implemented in percent, covering all six tasks; significant differences between the two groups at the level of 5% are asterisked *. On average test subjects took 7 minutes to access Help information when using Microsoft's Help system and 8 minutes when using atsolute's. This difference is not significant. When implementing the Help information, test subjects took an average of 1.5 minutes using the Microsoft Help system as opposed to 2 minutes when using that by atsolute. This difference is significant. Time taken over all tasks in minutes search Microsoft atsolute solving * Figure 5: Mean average time taken to access and implement help information in minutes, covering all six tasks; significant differences between the two groups at the level of 5% are asterisked *. When searching for information in the Microsoft Help system test subjects took an average of 16 steps in excess of the minimum required whilst the average when using atsolute's Help system was 26 steps above the minimum. This difference is highly significant. When implementing the information obtained using the Microsoft Help system test subjects took an

6 average of four steps as opposed to 3 steps after using atsolute's Help system. This difference is not significant. The number of steps taken beyond the minimum necessary to access and implement the information search solving * * Microsoft atsolute Figure 6: Number of steps activated when accessing and implementing help information, covering all six tasks; significant differences between the two groups at the level of 1% are asterisked **. After completing the tasks, test subjects were asked to evaluate their degree of satisfaction with the two help systems with regard to the tasks just performed. The diagram shown here is purely descriptive, the results of the examination of significance will be presented in the course of the symposium. Figure 7: Effect of the Help systems according to test subjects' subjective judgements based on a seven point scale from "definitely pro atsolute" to "definitely pro Microsoft". Overall, atsolute's Help system received the highest evaluation with regard to layout. On average test subjects stated that they tended to be more certain of what to click next when using atsolute's Help system and also that they found the time taken to complete the tasks as being too long when using atsolute. Test subjects tended to prefer using atsolute's product

7 stating that they found using Microsoft's Help system was more boring and that it also tended to make them more nervous. In spite of these advantages there was no evidence for a higher overall acceptance of the atsolute version. 5. Summary Contrary to atsolute's expectations, the results of the comparison demonstrate that the established Microsoft Help system performed better with regard to the usability criteria effectiveness and efficiency. It is surprising that despite these findings, atsolute's Help system achieved higher ratings for the majority of the satisfaction criteria considered in the study. One conclusion might be that test subjects preferred atsolute's individual design features (smaller range of choices, concise labeling and the use of graphic elements). In view of atsolute's inferior showing as regards effectiveness and efficiency it is perhaps not remarkable that the overall evaluation of the group of test subjects did not favor either Help system. The question arises as to the reasons responsible for the atsolute system's relatively poor performance. On the basis of observation and interviews conducted with test subjects, the following explanation appears probable: - The menu options which atsolute consciously chose to make concise were not sufficiently clear. Although atsolute had seen that this was in theory a possibility, the Bubble Help feature they provided was not fully developed and therefore failed to avoid areas of uncertainty. - The convincing idea of making navigational structure visible in the way that atsolute intended was not effected in such a manner as to make the "knowledge management tree" visible throughout navigation, but was restricted to showing just one visible section. These circumstances together with the fact the labeling of the graphics was at times ambiguous may have led to test subjects finding themselves unable to backtrack. Seen as a whole, the investigation should not be regarded as proof of whether the particular design features measure up to usability criteria or not. In order to answer this question it would be necessary to ensure in advance that the menu options are quite clear. In addition, a means of making the entire navigational path more transparent must be found. This could perhaps take the form of introducing a zoom feature, which users could move like a magnifying lens over the entire "knowledge management tree". References: atsolute GmbH (2003). DIN EN ISO (1999). Ergonomic requirements for office work with visual display terminals (VDTs); Guidance on usability. Jäning, J.S. (2003). knowledgetools - transforming information into knowledge. Microsoft Corporation (2003).