Kognitive Ergonomie: Vertiefung

Transcription

1 249 Class 10, June 29th, Class 10, June 29th, 2000 Thursday, June 29th, 2000 Methodology Case Study of Use of the GOMS Model (Slides from Class 8) Extended Example of an Empirical Evaluation Kognitive Ergonomie: Vertiefung Sommersemester 2000 Groupware Visualizing Conversation in Chat Circles (Slides from Class 6) General Organizational Obstacles WWW:

2 251 Evaluation of Two Library Interfaces Evaluation of Two Library Interfaces Introduction Introduction The HyperLib project have adopted a user-centred design methodology to ensure that user interfaces to library systems, developed on the project, support a wide variety of users. This is supported by an interactive process of (i) user requirements analysis, (ii) design and implementation and (iii) testing with representative users. As a result, it is believed that the project has developed an effective library OPAC search tool. This document reports on a series of user trials carried out at HUSAT to compare the usability of two interfaces to the library OPAC at the University of Antwerp. Both interfaces were accessible publicly (during November 1995) both to members of the university and to external users. One system employed a scrolling text (VT100) style interface and was based on Telnet access. The other was based on the World Wide Web using a set of hypertext linked pages. The two systems (referred to in this report as VT100 and WWW ) had similar OPAC functions enabling the user to search for library items according to a range of criteria such as: title, word from title, author, subject keyword, and subject number or UDC code. Both systems used the approach of inviting the user to select a method of searching, presenting them with a list of search results (author and title), and then allowing them to view items in more detail. The VT100 interface allowed the user to enter the numbers of items from the search list, to view further details, while the WWW interface allowed the user to view details by clicking on the items directly, or the checkboxes alongside them. In the latter case they could then step through the details of the items selected. With the VT100 interface, the user navigated through the system by entering menu numbers (or the initial letters of command options?). With the WWW interface, the user navigated by clicking on items displayed on screen in a traditional World Wide Web fashion. The study builds upon an earlier Hyperlib study which compared two VT100 text based library OPAC interface. The older of these two was used within this study as a baseline for comparison with the Web based system. While the other enhanced, VT100 interface provided extra facilities and flexibility, the results showed that its style of interface was not a particularly effective means of delivering these extra facilities to the user. Thus the WWW version was developed as an alternative delivery mechanism and so became the focus of this study. This material is taken directly from aa report that is available at the following URL: It is presented here as a detailed example of how an evaluation study can be designed and conducted

3 253 Evaluation of Two Library Interfaces Format of the Trials Format of the Trials Both versions of the user interface (VT100 and WWW) were tested in an experimental setting by 8 user subjects. They were required to perform a range of typical library search tasks whilst providing walkthrough comments following the approach of Polson et al (1992). (Note that the main difference between the method employed for this evaluation walkthrough approach and the verbal protocols conducted during the earlier Hyperlib OPAC comparison reported in deliverable 5.1 was that in this study, subjects were asked to step through the tasks and to consider and comment on the behaviour of the interfaces from a usability point of view. With the verbal protocol approach they were asked to give a commentary on their throught processes without necessarily making direct comments on the system itself.) From the user comments made during the walkthrough, and performance and subjective measurements made, it was possible to compare the two interfaces. Section 5 presents a description of the results obtained. In order to test and compre the two interfaces, subjects were asked to perform a set of library search tasks with each interface in turn. The two task sets comprised 8 equivalent tasks as used in the earlier comparison of VT100 interfaces (Deliverable D5.1). The task types, and the issues they addressed, are described in Table 1. Task Type Issue Addressed 1. Search on subject or author Test user s initial reactions to using specified by the user. system and ability of system to answer search question specified by them. 2. Search for book by author Ability of system to support a on a given subject. straightforward author search. 3. Find if same author has books Ability to make use of information on another subject and check received to continue with a search further details (either number on same author, as might occur in written or number available a browsing situation for borrowing).. 4. Find UDC code for a particular Ability of system to provide UDC subject to allow user to code to support searching browse shelves. on shelves. 5. To find a book on a specific Ability of system to allow user subject and of a specific to look for a book on a specific type (e.g. dictionary) or subject, but of a certain type. particular readership. 6. Search for ISBN number of a Ability of system to support book by a particular author the user who does know correct (whose name is misspelled). spelling of author s name. 7. To find out borrowing Ability of system to enable user conditions of a book, location to find details of code and borrowing as well as simply presence of book in number of copies available. library. 8. To find a book of a certain Ability of system to allow user type (e.g. dictionary or to look for a book covering one or encyclopaedia) for a friend more subjects and of a certain type. interested in a range of subjects, suitable as a present. Table 1: Task types employed for the trials and issues addressed

4 255 Evaluation of Two Library Interfaces Experimental Procedure Experimental Procedure Each user session was conducted as follows: 1. The subject was invited to sit down in front of the system terminal and asked to make themselves comfortable. This was seen as important in order to create a realistic setting for the trials. 2. They were then given an introduction to the project and the experiment and were told that they would be using two OPAC systems to access the Antwerp University Library. They were not told that one interface was a development of the other. Each subject was then given a brief verbal description of the facilities available to them via both interfaces, but told that they would be asked to use each version without any formal training (see Appendix 1). 3. The subject was then asked to complete a brief pre-test questionnaire to record their age group, course, current method of searching for books at Loughborough University, and use made of OPAC systems (Appendix 2). 4. The subject was then asked to perform the 8 specified tasks with one of the two interfaces, providing verbal protocols as they proceeded. The evaluator recorded their walkthrough comments and interaction events of interest on paper. The interactions and comments were also video recorded as a backup to note taking. Each subject was told that they should proceed at a comfortable pace and that they would be timed. They should try to complete each task to their own satisfaction (as in a real search situation) but if could stop and move onto the next if they felt that they could not finish it. 5. Having completed each set of tasks, the subject then completed a post-test questionnaire (Appendix 3), for which they gave ease/difficulty ratings and supporting comments on a range of different usability aspects. The user was also asked to state whether they would use the system on a regular basis and to provide supporting comments. 6. The same procedure was repeated with the second system and the second set of tasks, followed by completion of the same post test questionnaire. 7. Having used both interfaces, each subject was asked to state which of the two they would prefer to use and to provide written comments comparing them. (See Appendix 4). 8. Finally the users were thanked and given payment for their time.

5 257 Evaluation of Two Library Interfaces Walkthrough Comments Walkthrough Comments The main method of data capture for this experiment was that of user walkthrough comments and evaluator observations. During each session, evaluator notes were taken of these events while a video recording was also made to clarify events and fill in gaps. From these data it was possible to identify design strengths and weaknesses of both interfaces. Particular note was taken of user problems that were experienced by more than a single user subject. As a result of the protocol analysis and user interactions, a number of key user interface issues regarding the WWW system were highlighted, including both well received features of the interface, and areas where problems were encountered. These diagnostics are presented in Table 2 below: Positive Aspects Problems 1. Attractive presentation. 1. Use of the system on Made good layout use of a a standard sized screen meant graphics screen. that the user had to scroll to see all the information and buttons on one screen. This sometimes meant that some buttons were not noticed. 2. Clicking with a mouse seen 2. Check boxes useful but not as a simple and natural means well explained so few users of selecting items to view made use of them. details and navigating through the system. 3. Interface made it easy to 3. No Boolean search facilities. follow structure of system and to navigate around it. 4. References generally laid out 4. No loan information available as in a spacious, understandable with VT100 interface. and consistent manner. 5. Response times generally good. 5. No instruction on how to enter author names. 6. Menu driven approach seems 6. Desire to enter two or more to be easily understood keywords and words in title e.g. by users. pop art or non parametric. System seems only to accept first word. 7. Not case sensitive to user 7. Not all books have UDC code inputs. (probably due to different data format adopted by the different libraries). 8. Flexible range of search 8. No partial match facility facilities easily selected from top level menu. to cater for spelling errors. However the display of a list of names did help to locate correct name. 9. Search results heading not always noticed, so not always realised when results were split over two or more screens. 10. Expectation that View all would display all titles, not the details of one title. Table 2: Positive aspects and problems of the WWW interface.

6 259 Evaluation of Two Library Interfaces Task time Task Time This is the total time taken (in minutes) by the user for each of the 8 tasks, excluding time spent in reading the tasks from the sheet before starting them. The mean task time for each user was then calculated across all 8 tasks. Finally, a mean task time across all subjects was calculated. User VT100 Interface WWW Interface Mean Table 3: Mean task time (the lower the better) Table 3 shows that shorter task times were obtained with the VT100 interface. However comments made during the sessions indicated that users were encouraged to explore more with the WWW interface and so this explains, to some extent, the longer search times

7 261 Evaluation of Two Library Interfaces Task achievement Task Achievement This measure is an estimate of the amount of each task completed. Clearly this is important since the effectiveness of the system is based on its ability to produce useful search results. A number of rules of thumb were adopted for making this estimate. A simple categorisation was adopted as shown in Table 4. Achievement Rating Exact items required obtained 100% Minor detail incorrect e.g. not all copies of a book located 75% Reasonable results obtained but not exact result expected e.g. book found on similar subject 50% Good attempt at search but results only loosely related to target 25% No results obtained 0% Table 4: Categories for rating task achievement Using this scoring scheme, the average task achievement level for each user was then calculated across all 8 tasks. Finally, a mean task achievement level across all users was calculated. User VT100 System WWW System Mean Table 5: Mean task achievement (the higher the better) The VT100 interface allowed users to achieve slightly better results in terms of the quality of the search output as shown in Table 5. This slight difference may partly be due the WWW interface not providing all the information options (e.g. loan information) that were present via the more mature VT100 interface.

8 263 Evaluation of Two Library Interfaces Task performance rating Task Performance Rating This measure is a combination of measures 1 and 2 and is based upon: Task achievement (%) / Task time (min) Thus it represents the ability of the user to obtain search results in a short time i.e. it is a form of efficiency measure. User VT100 System WWW System Mean Table 6: Mean task performance rating (the higher the better) As can be seen, the VT100 interface produced superior task performance ratings as shown in Table 6. This is mainly due to shorter search times rather than to better results being obtained.

9 265 Evaluation of Two Library Interfaces Results of Subjective Data (1) Results of Subjective Data Following the completion of a set of tasks with each of the two interfaces, each subject was invited to complete a post-test questionnaire (see Appendix 3). They were asked to consider 8 different usability factors for the interface in turn, to give it a rating on a 6 point scale, and then offer any supporting comments. A summary of the results obtained for each factor for the WWW interface is presented below, supplemented with comments made during the walkthroughs. For comparison, some of these comments are contrasted with the responses given for the VT100 interface. The full details of the subjective data captured for both interfaces can be found in Appendix 7. (Please note that for simplicity, each interface was referred ti as a system although subjects were made aware that a common database was being used but accessed via different interfaces). 1. System Instructions Question:Were the instructions, options and commands for using the system easy to follow and use? The instructions for the WWW interface were regarded as perhaps too sparse and more information on what the user needed to do could be given at each stage of the dialogue. In contrast the instructions for the VT100 interface, while not optimal, were employed by users in the trial to understand how to use the OPAC system. A little more information could be given for some screens on the WWW interface particularly to highlight some little used features such as the multiple selection of items, resulting from a search, using checkboxes (which few subjects used). The help facility was referred to by several of the users but they tended to feel that it was too detailed and long winded. 2. Layout and presentation of information Question: Was the layout and presentation of information clear? The layout and presentation of the WWW interface was regarded as attractive and encourage use. However it was commented that too much space was used for the graphics at the top of the screen so that on a standard 13" Apple screen, some important controls were not immediately visible. It was also necessary to scroll down to view some of the instructions. The listing of search results on screen (title and author) was seen as slightly cluttered on the WWW interface (although clearer than on the VT100) system and users occasionally failed to spot the book they were looking for. However the layout of individual book details was thought fairly clear via the WWW interface, and an improvement on the VT100. The other problem identified was that the status bar indicating the number of items found was not always seen by users who occasionally failed to realise that the list went over two or more pages. 3. Ease of navigation Question: Was it easy to move around different parts of the system? The mechanism for navigation through the WWW interface was found to work quite well, although some users failed to find the facility to return directly to the initial screen. There was some confusion between the back key and the WWW browser s step back facility as to whether they returned the user to the previous screen or up to the previous level in the hierarchy. Perhaps a clearer description of the OPAC back key would be useful. However, in general, stepping back caused far fewer problems than with the VT100 interface as users seemed to have to experiment to find out how to achieve this. With the WWW interface, the subject navigation tool was regarded as an attractive option but users frequently failed to find the subject they required. They also found that the listing of broader and narrower terms was not clear and some commented that they found the process circular (e.g. by selecting the broader option while thinking they were narrowing down). 4. Flexibility Question: Did the system allow you sufficient flexibility to work in the way you wanted? The WWW interface did allow subjects to specify a second criteria after obtaining the results of an initial search, however no one seemed to use this facility indicating that it was not clearly presented. There was a general feeling that users wanted to be able to specify two keywords (possibly with simple Boolean links) or both names of an author. This was not available via either the WWW or the VT100 interface. The subject search was thought to require modification by users who were unable to find specific subjects. Having obtained a list of search results, users liked the ability to jump ahead to, say, the set of authors whose surname began with a certain letter. However the placement of the Jump to button to the left of the entry field meant that they sometimes pressed the button before entering the letter. In general users attempted searches via a number of different routes thus making use of the flexibility

10 267 Evaluation of Two Library Interfaces provided by both interfaces. Results of Subjective Data (2) 5. Helpfulness in coping with errors Question: Was the system helpful in coping with any errors that were made? Both interfaces supported the user by allowing them to reverse and retry a command if not understood. However the actual level of help given to overcome an error was seen as limited. The WWW interface did, however, provide a list of items in response to a search term. Thus if, say, an author s name was misspelled it occasionally appeared within the subsequent list. As previously stated, the help facility was seen as too lengthy and so not really appropriate for assisting with errors. 6. System Responsiveness Question: Did you find the system responsive to your inputs? Regarding actual response times, both interfaces were thought to provide reasonably quick responses, although this slowed slightly on occasions, probably reflecting the current load on the host computer. The WWW interface provided feedback that it was performing a search which was an improvement over the VT100 system. 7. Suitability for Searching Question: Do you consider the system suitable for the task of searching through a library catalogue? Both interfaces were seen as providing some good basic search facilities. Several users thought that they would make better use of the full range of WWW facilities when they became used to them. Some minor enhancements were still needed such as providing information on loan details which one user stated that she always wanted to know about when using an OPAC. It was also thought that the UDC and ISBN codes should be provided for each book listed. 8. Regular use of the system? Question: Would you want to use this system on a regular basis? For the VT100 interface, only a minority of users thought they would use the system regularly (2 out of 8) while the majority (7 out of 8) stated that they would use the WWW interface on a regular basis. In general, the WWW interface was regarded as friendly and attractive ( easy on the eye ). It was perhaps slightly tedious in places e.g. the constant need to scroll down when using a standard screen size, and the subject search facility could be enhanced to match user expectations and a wider range of subject names in some way. 9. Usability Factor Ratings Usability Factors VT100 Interface WWW Interface Instructions, options and commands Layout and presentation Navigation around the system Flexibility Helpfulness in coping with errors Response times Suitability for search task Overall Mean Rating Table 8. Mean usability factor ratings A summary of the usability factor ratings is presented in Table 8, which shows that for each factor, subjects rated the WWW interface much more positively and all but exceeded the 3.5 score (on a 0-5 scale). This bodes well for the success of the new WWW facility. 10. Preferred interface Finally the user was asked to compare the two interfaces they had used. Question: Of the two systems, which would you prefer to use? Six of the subjects indicated that they preferred the new WWW interface in preference to the VT100 interface.

11 269 Evaluation of Two Library Interfaces Discussion Discussion The results of the study showed that overall the WWW interface was rated very positively in comparison with the VT100 system. In fact 6 out of the 8 subjects preferred the WWW version, finding it more attractive in appearance, and easier to navigate and to recover from error. However it was thought that the layout of information could be refined and the help system simplified to support the inexperienced user. The WWW interface offered more search facilities such as the ability to refine a search by a further criterion and to navigate through a subject structure. These additional features were appreciated by the subjects. The results showed however, that they need to be reviewed to make sure that they are easy to access and to navigate. In terms of performance, users of both interfaces were able to achieve roughly equivalent results, but surprisingly they achieved these in a shorter time with the VT100 version. This was thought to be due to the fact that users spent longer exploring the different facilities available through the WWW version. As one user stated, it encouraged them to explore. The usability of the WWW interface could be further improved by addressing the main comments made during the walkthroughs as highlighted below: The need to support searching on two keywords or full author names. Review layout of buttons and button lables. Improve layout and status information. Provide current loan information. Provide briefer help information and improved instruction. As stated in D5.1 (The Evaluation of the HyperLib Interfaces), there are considerable constraints imposed by the rigid dialogue structure of the VT100 environment. The report also highlighted the fact that the provision of search enhancements increased the complexity of the interface, and concluded that, in the context of the VT100 environment, the goals of producing easy to use and powerful interfaces are to some extent opposed to each other. In contrast, the GUI style of interface adopted for the WWW version was found to increase the flexibility of search based activities. Given that the enhanced VT100 and WWW OPAC interfaces share very similar search functions, the positive results for the latter suggests that they are mainly due to the WWW environment itself. This appears to indicate that the WWW version of the OPAC interface represents a clear step forward towards improving access to library catalogue systems for a wide range of library users.

12 271 Evaluation of Two Library Interfaces Conclusion Conclusion The user-based evaluation of the two OPAC interfaces, reported in this document, shows the effectiveness of a graphical style of interaction compared with the traditional text-based approach. The ability to select the search strategy andto get more details on items retrieved by clicking with a mouse was found easy, even for inexperienced users. In particular the ability to reverse an action gave users the confidence to explore facilities and not worry about mistakes. With a little further enhancement, the WWW interface tested in the study would go a long way to meeting the needs of library OPAC users.

13 273 Evaluation of Two Library Interfaces Introduction Given to Subjects Appendix 1: Introduction given to subjects Introduction to the OPAC study The aim of this set of trials is to test two versions of a system for accessing a catalogue of books and journals from the library at Antwerp University. Both have similar facilities: To search by author To search on a title To search on a subject keyword. You will be given a series of tasks to perform with each system in turn but remember, it is the system that is being tested, not you. After you have performed each of these tasks, you will be asked to provide comments on the system. These will be recorded on tape for checking purposes at the end. You will not be timed in performing the tasks so just proceed at a comfortable pace without hurrying. Please ask if you have any questions.

14 275 Evaluation of Two Library Interfaces Pre-Test Questionnaire Appendix 2: Pre-test questionnaire OPAC trials - Pre-test questionnaire Subject name:... Number:... Age range: o o o 60+ o Course: (Type of degree and area of study) When you use the library at LUT how do you normally search for books or articles? Have you used a computer based bibliography or OPAC before? No o Yes - once or twice o Yes - sometimes o Yes - often o What types of use to do make of it?

15 277 Evaluation of Two Library Interfaces Post-Test Questionnaire (1) Appendix 4: Post-test questionnaire OPAC Trials - Post test questionnaire System: 1 / 2 Tasks: A / B Circle as appropriate Please rate the following aspects of the system you have just used: 1. Were the instructions, options and commands on using the system easy to follow and use? Very easy Very difficult If not very easy then please give details: 2. Was the layout and presentation of information clear? Very clear Very unclear If not very clear then please give details: 3. Was it easy to move around different parts of the system? Very easy Very difficult If not very easy then please give details: 4. Did the system allow you sufficient flexibility to work in the way you wanted? Very flexible Very inflexible If not very flexible then please give details: 5. Was the system helpful in coping with any errors that were made? Very helpful Very unhelpful If not very helpful then please give details: 6. Did you find the system responsive to your inputs? Very responsive Very unresponsive If not very responsive then please give details:

16 279 Evaluation of Two Library Interfaces Post-Test Questionnaire (2) 7. Do you consider the system suitable for the task of searching through a library catalogue? Very suitable Very unsuitable If not very suitable then please give details: 8. Is this a system you would want to use on a regular basis? Yes / No Do you have any other comments about the system? Comparison Questionnaire 280 Appendix 5: Comparison questionnaire After using both systems: Of the two systems, which would you prefer to use? First used / Second used Comments:

17 281 Evaluation of Two Library Interfaces Task Sets A and B (1) Appendix 5: Task sets A & B Tasks - Set A 1. First of all, write down the name of a subject or author of particular interest to you (a new name if this is the second time you are performing this task): Now try to a book on this topic that you would find of interest and briefly write down the name of it. 2. You have heard that Isaac Asimov has written a book on Halley s comet. Check if it is in the library and if you find it, write down the title and location code? 3. Now check if Isaac Asimov has any books in the library from his Foundation series of books. If so write down the number of items in this series that the library keeps? 4. You want to find the UDC code on hazardous substances so that you can browse through the shelves on that and similar topics. Try to find a book on this subject and write down the code? 5. You are looking for a book on pop art, preferably a dictionary. If you find one that matches the description, what is the publisher s name and year of publication? 6. You receive a telephone message to find a book by Sidney Siegell on nonparametric statistics. What is the ISBN number of this book? 7. For the book Fishes of the World, check and write down how many copies have not been lent out? 8. You have a friend interested in music, decor and dolls. You want to buy her an encyclopaedia on one of the topics but with not too many pages. Find a suitable book to buy for them? Tasks - Set B 1. First of all, write down the name of a subject or author of particular interest to you (a new name if this is the second time you are performing this task): Now try to a book on this topic that you would find of interest and briefly write down the name of it. 2. You have been told of a book by an author called Donald Norman about the ease of using everyday products. Check if it is in the library and if you find it, write down the publishers name and year of publication?

18 283 Evaluation of Two Library Interfaces Task Sets A and B (2) 3. Now find out if Donald Norman has written any books on human-computer interaction. If so write down the number of books or articles he has written? 4. You want to find the UDC code on books about birds of Africa, so that you can browse through the shelves on that and similar topics. Try to find the code and write it down. 5. You are looking for a book on lateral thinking, preferably for business people. If you find one that matches the description, what is the author s name and year of publication? 6. Try to find a book by Ben Shneidderman on Human-Computer Interaction. If you can write down the ISBN number for it. 7. Find out how many copies of the book Grays Anatomy have not yet been lent out from the Leeszaal library? 8. You have a friend interested in superstitions, Chinese proverbs, aerospace. You want to buy him a dictionary on one of the topics but with not too many pages. Find a suitable book to buy for them.

19 285 Evaluation of Two Library Interfaces Task Performance Data (1) Appendix 6: Task performance data The following tables show the results obtained for both the VT100 and WWW interfaces across all 3 measures. To interpret the results it should be noted that: The following tables show the results obtained for both the old and new interfaces across all 3 measures. To interpret the results it should be noted that: Measure 1 - Task time: the LOWER the time the better. Measure 2 - Task achievement: the HIGHER the score the better Measure 3 - Task performance: the HIGHER the rating the better. Task Times (Minutes) User Task Task Task Task Task Task Task Task Total Mean Mean 4.07 Table 9: Task times for the VT100 interface User Task Task Task Task Task Task Task Task Total Mean Mean 5.82 Table 10: Task times for the WWW interface Task Achievement (Percentage of Task Achieved) User Task Task Task Task Task Task Task Task Mean Mean 76.7 Table 11: Task achievement for the VT100 interface User Task Task Task Task Task Task Task Task Mean

20 287 Evaluation of Two Library Interfaces Task Performance Data (2) Mean 72.9 Table 12: Task achievement for the WWW interface Task Performance Rating User Task Task Task Task Task Task Task Task Mean Mean Table 13: Task performance rating for the VT100 interface User Task Task Task Task Task Task Task Task Mean Mean Table 14: Task Task performance rating for the WWW interface

21 289 Evaluation of Two Library Interfaces Subjective Data (1) Appendix 7: Subjective data The following tables group the replies given for each factor in the post-test questionnaire (see Appendix 3). Each table represents a single factor with comments made by each user (U1 - U8), their individual ratings, together with the mean ratings. System Instructions Question: Were the instructions, options and commands on using the system easy to follow and use? VT100 Interface WWW Interface U Rating Comments Rating Comments 1 4 Would improve with 5 familiarity but options and commands quite limited so even if unfamiliar, can always get back to original screen by escaping (although it does not tell you this). 2 1 Instructions not 5 clearly shown. Constant instructions, not just in help, would be good. Nothing to guide you through the hierarchy quickly. 3 4 Keyword still only 2 Once past the initial screen, gives 1 word option it was very difficult (as with other system). to find specific things. More would be better. Also there do not seem to be any options within the headings e.g. if you want a music encyclopaedia, you have to look through all 400+ articles on music. 4 1 The only reason I 5 Although sometimes slightly managed was because I confusing it was easy to had used similar rectify any errors. systems on campus. Some functions were missed A first time user may at the bottom of the struggle for a while. screen - did not see them. Instructions were vague, not stereotypical, i.e. enter goes back rather than forward, and it was hard to manoeuvre 5 1 Was not made clear on 4 Was not explicitly clear how which keys could be to use options and commands. used to help navigate the system. Instructions and comments not explicit 6 2 Yes they were 4 Yes they were easy to reasonably easy to use because they were use. Would be quicker iconed on the screen. with a mouse The main heading were easy to follow but getting specific information at times was very difficult. 8 3 Difficult to find 2 Terms confusing. I expected

22 291 Evaluation of Two Library Interfaces Subjective Data (2) the option you wanted, mix of letters, numbers and cursor key. VIEW ALL to give me a complete list of the author s books when it just gave me details on the book the mouse was on. 2.6 MEAN SCORE 3.75 MEAN SCORE Table 15: Ratings and comments on system instructions Layout and presentation of information Question: Was the layout and presentation of information clear? VT100 Interface WWW Interface U Rating Comments Rating Comments 1 0 Too close together, 5 not enough space between information and choices. Lack of mobility in accessing/making choice. 2 1 Hard to understand 4 some choices as not in English. Not sure what they were trying to achieve with black bars. Information in them not clear and constant use of capitals put me off Options, once you find the book are unclear. What do things like view all mean? Also little information about what actually appears in books i.e. summary would be helpful. 4 1 No, information was 4 Go back one step and not descriptive back 2 different functions enough which are not clearly explained. Subject search function was harder to navigate, restrictive and confusing at time. 5 0 The was not structure 4 A lot of information to the text was on-screen. (columns/rows) etc. making it difficult to make sense of it. 6 1 If the keys that you 5 can use for different routings were listed at the top of each page, it would be easier to use HELP information too clutter and detailed - needs clearer quick access headings 8 1 Very close together, 4 same font throughout, no separation of title MEAN SCORE 4 MEAN SCORE Table 16: Ratings and comments on layout and presentation

23 293 Organizational Problems for Groupware 294 Organizational Problems for Groupware 293 List of Problems 1. Distribution of effort and benefits Who has to invest more effort, and who benefits most from such effort? 2. Difficulty of predicting patterns of system use It is more difficult for a designer to imagine how S will be used on the basis of her own exerience and intuition 3. Complications involved in evaluation A number of factors make groupware systems generally more difficult to evaluate than single-user systems These problems, along with several others, were originally formulated by Grudin in several influential articles (see the references in the textbook). Problems With Specific Types of System (1) 294 Shared diaries 1. Distribution of effort and benefits 2. Difficulty of predicting patterns of system use 3. Complications involved in evaluation

24 295 Organizational Problems for Groupware Problems With Specific Types of System (2) Discussion forums 1. Distribution of effort and benefits 2. Difficulty of predicting patterns of system use 3. Complications involved in evaluation Problems With Specific Types of System (3) 296 Group meeting rooms 1. Distribution of effort and benefits 2. Difficulty of predicting patterns of system use 3. Complications involved in evaluation

25 297 Organizational Problems for Groupware Problems With Specific Types of System (4) Remote conferencing systems 1. Distribution of effort and benefits 2. Difficulty of predicting patterns of system use 3. Complications involved in evaluation Reading Reading for Class Chapter 11 - Evaluation Techniques Overview 11.1 What is evaluation?, p Goals of evaluation, p Styles of evaluation, p Evaluating the design, p Evaluating the implementation, p. 415 Exception: Pp , concerning statistical measures, do not have to be read before Class 11 [End], p. 436