Learning from LEARN usability testing of the University of Auckland Library Website

Transcription

1 Learning from LEARN: Usability Testing of the University of Auckland Library Website Chidlow, R, Coates, J and Wilkinson, E The University of Auckland Library, Auckland, NZ Acknowledgements: Hester Mountifield, Li Wang; members of the LEARN Advisory Group. Abstract The University of Auckland Library has over 500 databases, 180,000 electronic books, 58,000 electronic serials and that's just the start of it. To access this information students and staff of the university must use LEARN, the Library website. It's all on LEARN, but can they find it? Following a planned upgrade to the Library's website, we wanted to know how people actually used LEARN before we made any changes. This meant a usability test. The results were surprising oddly enough people did not use the website the way we expected them to! The data collated from the usability testing was then incorporated in the new design of LEARN and, just to make sure we got it right, following the trends in current literature on the subject, we undertook another usability test to see whether our new design was easier to use than the previous incarnation did we get it right? Introduction In 2004 the University of Auckland Library undertook usability testing of its website, LEARN, in association with a planned upgrade. 1 A steering group oversaw the exercise, which was initiated in order to gather user-generated information, placing patrons squarely within the redesign process. How were users navigating the existing page; did they understand the terminology; what were their expectations and assumptions; how was the page really being used? Was LEARN as efficacious as Veldof argues websites should be: Users should not have to think about using the interface they should be thinking about the content. 2 The objectives of the test programme were to: incorporate user feedback into the web design process specifically inform design of the LEARN front page i.e. primary target is use of 1 st level links and menus Implicit in the programme was an assessment of the worth of usability testing as an evaluation tool. Usability testing is a qualitative method of evaluation; inferences are drawn from close observation of how a site is used in real situations, rather than by applying statistical tools. It is based on the premise that people don t necessarily do what they say they do. This form of testing has proved particularly suitable for evaluating website design, and has been used quite extensively over the last decade. 1 LEARN stands for Library Electronic Academic Resources Network 2 Jerily R. Veldof, "Usability Testing," in Developing Web-Based Instruction : Planning, Designing, Managing and Evaluating for Results, ed. Elizabeth A. Dupuis (New York: Neal-Schuman Publishers, 2003). p. 145 Page 1 of 19

2 Usability testing of the LEARN homepage was undertaken in October 2004 and an analysis of the results fed into the review process. Modifications were made accordingly and LEARN s new interface was launched in January Retesting to assist in evaluating the changes took place the following June; findings and comparative results were reported in July The testing programme opened a window on our users as they navigated LEARN, and modifications were made to reflect some common behaviours. General insights into users preconceptions, search strategies and expectations were also gained: staff involved were intrigued, frustrated and energized. Background LEARN was launched nine years ago in 1996, and has undergone several transformations in that time. In 2004 it was decided to rebrand the whole University website and a concurrent review of the 2002 iteration of LEARN (current at the time) seemed appropriate. (See Appendix 5) The LEARN Advisory Group (LAG) was set up to: provide strategic focus to LEARN provide ongoing monitoring act as a forum manage the review process Members included staff from the Library s Digital Services, Learning Services, and various Information Services teams. Following past practice, comments and suggestions from Library staff were called for and members of LAG began detailed reviews of various aspects of the homepage. During early discussions it was noted that available user data was limited to hit statistics; a single puzzle piece rather than the whole picture. Advocates of usability testing made the point that the search behaviours and preferences of students and staff were of acknowledged importance, but largely unknown. Were people using LEARN as expected and assumed? In addition, the University s teaching and learning focus was shifting towards being student-centred. The Library has a key role to play in that arena, as well as supporting research. Undertaking usability testing would be one way of addressing these issues. Three members of LAG were joined by two from Information Skills to implement this exercise. Usability testing was undertaken in October 2004 and again in June 2005, subsequent to LEARN s redesign. This evaluation tool was new to the University Library and so the testing team had to establish principles and procedures from the outset. Usability and usability evaluation Usability is, most simply, the degree to which something is useful, and the concept has been applied to many technologies. It has become increasingly important to interface design and information architecture since amateurs became significant within user populations. A core principle of usability studies has been the emphasis on the end-user. An extensive literature has developed from the 1990s, and a number of papers related to the usability of library web sites have been published in the last eight years. A user-centred approach to the design of library interfaces has been described by Veldof as a significant shift. Page 2 of 19

3 Many library Web sites have been developed behind closed doors with input only from librarians and technical staff. The results are often text-heavy, jargon-filled sites that require users to have prior experience with libraries and online research. Ultimately, these Web sites create barriers between the audience and the content, increasing users frustration and failure rates. 3 It was challenges such as these which influenced the LEARN Advisory Group s decision to evaluate the Library homepage using direct input from users, alongside other means of review. Evaluations of usability encompass both qualitative and quantitative methodologies. They can be divided into three types: enquiry, inspection, and formal usability testing. 4 Enquiry methods ask users to describe their behaviour or potential behaviour, by means of surveys, interviews or focus groups. Difficulties with these include expense, peer pressure and the probability that people often don t behave as they believe or say they do. 5 Inspection is undertaken by experts web designers or library staff and may involve heuristic evaluation (establishing rules of thumb) and cognitive walkthrough (thinking through step by step). One aspect that should be considered with these methods is provider capture. Veldof et al propose a librarians mental model. 6 Rubin mentions inside information and vested interest in the site s success. 7 None of these considerations negates the usefulness of consulting those in the know, but evaluation could become unbalanced or misleading without input from end-users. Formal usability testing involves setting representative users real tasks, observing their behaviour, and drawing conclusions. The goal is to identify and address deficiencies in such a way that ease of use is maximised. Usability testing is a research tool which can vary in complexity, but a number of principles can be applied. Dumas and Redish 8 propose that: the goal is to improve the usability of the interface testers represent real users testers perform real tasks user behavior and commentary are observed and recorded data is analyzed to recognize problems and suggest solutions Usability testing: group size In recent years there has been a trend towards using quite small groups for qualitative usability testing. For those accustomed to the statistically significant sample sizes associated with survey methods, the number of people being tested (as low as five) may appear unusual, even suspect. 3 Ibid. p Brenda Battleson, Austin Booth, and Jane Weintrop, "Usability Testing of an Academic Library Web Site: A Case Study," The Journal of Academic Librarianship 27, no. 3 (2001). p. 189,90 5 Susanna Davidsen and Everyl Yankee, Web Site Design with the Patron in Mind : A Step-by- Step Guide for Libraries (Chicago: American Library Association, 2004). p Veldof, "Usability Testing." pp. 129, Jeffrey Rubin, Handbook of Usability Testing : How to Plan, Design, and Conduct Effective Tests (New York: John Wiley & Sons, Inc, 1994). p Battleson, Booth, and Weintrop, "Usability Testing of an Academic Library Web Site: A Case Study." p. 189 Page 3 of 19

4 However Neilsen argues that 30% of gross usability problems become apparent with just one user and that five users will uncover the majority of issues. 9 He advocates iterative testing of groups of five homogeneous users so that resources can be spread over the project. If the user population is heterogeneous, the drafting of more participants is advisable. Figure 1 Nielsen, J. (2000). Selected studies of usability tests undertaken by libraries and database providers used between 10 and 134 participants; most used 25 or fewer. In the case where 134 participated, researchers concluded that the size of the group was an unnecessarily resource-intensive exercise and the same conclusions could have been reached using much smaller, targeted groups. 10 LEARN usability testing: methodology The testing team s programme began with a briefing session at which the principles of usability testing were outlined. The test session structure was designed; tasks formulated; and a procedure for facilitators agreed upon. (See Appendices 1 and 2) Each test consisted of: six general questions to provide background information and act as a warm-up ten short tasks, based on selecting appropriate links from the LEARN page five discussion questions open-ended comment. It was anticipated that each one-on-one test would take up to an hour. Student sessions took place in computer training rooms; academic and general staff were visited in their own offices. It was emphasised to all participants that it was the library website being tested, not them. Nevertheless, some people were nervous and facilitators had to be prepared to reassure them on the day. 9 Jakob Nielsen, Why You Only Need to Test with 5 Users (2000 [cited 24th August 2004]); available from 10 Janice Krueger, Ron L. Ray, and Lorrie Knight, "Applying Web Usability Techniques to Assess Student Awareness of Library Web Resources," The Journal of Academic Librarianship 30, no. 4 (2004). p. 292 Page 4 of 19

5 The participants comprised students, general staff and academics, who were invited to take part. Amongst the group there was a spread of ability, subject speciality, familiarity with the Library and time associated with the University. Twelve people took part in the first test and eleven in the second. The tasks section is key to usability testing. Ten tasks were designed to test important links on the LEARN homepage. e.g. for Voyager, the catalogue: You want to find out if the Library has The Lord of the Rings. Where do you go to find it? Although participants were encouraged to complete the tasks if possible, the primary goal of the testing was to ascertain whether appropriate LEARN homepage links were selected. Participants were given one task at a time, in random order, and encouraged to vocalise continuously as they attempted the task. Continuous vocalisation is known as think aloud protocol. 11 The facilitator observed very closely, taking detailed notes of navigational actions and user commentary. It was important for facilitators to refrain from giving guidance and direction, and to keep comments minimal and as neutral as possible. [Facilitators could provide helpful advice later on; we didn t allow our users to leave unelightened!] The redesigned LEARN page was tested in June 2005 using exactly the same methodology. The same people acted as facilitators but other participants were approached and broadly matched the profile of the first group. LEARN usability testing: results 2004 Usability was assessed by degree of success and time taken to locate and settle on an appropriate link. Four categories were established and a team approach taken to assessing the transcripts by task: Succeeded with ease / found appropriate link quickly Succeeded with difficulty / found appropriate link after some time; occasionally stumbled across the correct answer Failed / an appropriate link was not found Misinterpret a few participants failed to understand one or more questions. First Test Results (See Appendices 3 and 5) From this exercise, the LEARN links which were hardest for users to identify as solutions to tasks became apparent. Tasks undertaken with most ease included those targeting: Voyager (Catalogue) About the Library (Hours) Ask a Librarian Tasks undertaken with difficulty included those targeting: Document Delivery Library Course Bookings 11 Battleson, Booth, and Weintrop, "Usability Testing of an Academic Library Web Site: A Case Study." p. 191 Page 5 of 19

6 Tasks where failure was common included those targeting: Databases (Articles on a topic) About the Library (Borrowing limits) Training and Help (Referencing styles) Contacts (Subject Librarian) Tasks where question was (also) misinterpreted: Ask a Librarian Resources by Subject (information on a subject) LEARN usability testing: analysis and implementation On summarising test results for LAG, attention was drawn to links where difficulty and failure had featured. These were given particular consideration during the remainder of the review process. Alterations made to links on the LEARN homepage were a combination of re-organisation, rewording and positional changes. Usability testing also influenced decisions about retention, addition and deletion. Another factor influencing the redesign were University re-branding parameters. These included colour and header/footer requirements. (See Appendix 6) Re-organisation: LAG had already been considering grouping links in columns under headings such as Resources and Services. Usability testing confirmed this as potentially useful; some participants making a direct suggestion. Re-wording: Links were reworded as follows: Databases to Database and Article Searching Document Delivery to Inter-Campus Delivery (ICLDS) Request and Interloan Request About the Library to Libraries, Services, Collections Library Course Bookings to Book a Library Course Contacts to Library Contacts Re-Positioning: Aside from assigning links to Resources or Services, the following are of note: Library Course Bookings: Users often thought this link was part of the central graphic; they did not see it as a clickable link Contacts: The 2002 placement of this link with others not directly related to patrons use of the Library (Intranet, Vacancies etc) acted to obscure it or infer it was a University list Links not connected to patrons use of the Library were repositioned outside the main frames or removed e.g. University homepage, Intranet, Vacancies. Retention, Addition, Deletion: Retention of the I need to drop down menu is of note. Usability testing revealed that this feature was used more heavily than had been thought, and was regarded by patrons as very useful. Options within the menu were reviewed and reorganised In an attempt to elucidate Document Delivery the link was not only reworded, but split. Page 6 of 19

7 Discussion around solutions necessarily covered possible reasons for users ineffective decisionmaking, and it was noted that conceptual difficulties played a significant role. As a result some work was also done on secondary pages such as that following the E-journals link often used erroneously. LEARN usability testing: results 2005 (See Appendices 3 and 6) User behaviour on tasks was categorised similarly. Tasks undertaken with most ease included those targeting: Voyager (Catalogue) Libraries, Services, Collections (Borrowing limits) Library Hours Book a Library Course Tasks undertaken with difficulty included those targeting: Database and Article Searching Inter-Campus Delivery (ICLDS) Request Library Contacts (Subject Librarian) Training & Help (Referencing) Task where failure was common: Resources by Subject (information on a subject) Tasks where question was (also) misinterpreted: Ask a Librarian Resources by Subject Training & Help (Referencing) Comparison LEARN 2002 and 2005 For tasks targeting: article searching, borrowing limits, booking library courses and contacting a Subject Librarian the success rate improved between the first test and the retest. Relevant Links: Database and Article Searching; Libraries, Services and Collections, Book a Library Course, Library Contacts Voyager catalogue, document delivery and hours there was minimal or no change between both tests, users succeeded at similar rates. Relevant Links: Voyager, Inter-Campus Delivery (ICLDS) Request, Library Hours finding information on a subject and accessing library assistance remotely saw a decrease in success rate between the two tests. Relevant Links: Resources by Subject; Ask-a-Librarian finding information about referencing the failure rate reduced, but the number of users for whom this task was completed with difficulty increased. Relevant Link: Training & Help Page 7 of 19

8 LEARN Usability Testing: Discussion It was not difficult to identify major areas of concern from the initial testing, and some of them reinforced impressions of user behaviour already held by Library staff. Links which were modified and showed improvement included [2005 wording]: Databases and Article searching Libraries, Services and Collections Book a Library Course Library Contacts Overall improvements to usability have stemmed from the top-level organisation into Resources and Services, the revamp of I need to menu and the removal of some tenuously relevant links. Many participants commented favourably on the clean layout of LEARN Areas that remain of particular concern include: Inter-Campus Delivery (ICLDS) Request Databases and Article searching The wording of the Document Delivery link was much discussed, and it is not apparent that an ideal state has been reached by rewording and splitting the access point. Part of the difficulty stems from students ignorance of the services and their preconditions, which is compounded by terminology. Conceptual difficulties became very apparent when users were asked to find journal articles on a subject; the task associated with the Databases link. Terminology played a part here too: some students becoming confused between journals, serials and articles. However the underlying problem was usually a low level of comprehension regarding the top left-hand links on LEARN: Voyager (catalogue), Databases, E-Journals. People showed real difficulty in deciding which of these to use. Even when article searching was added to the Databases link more students than not exhibited confusion over this task. The content, relationships and functioning of these tools is not widely understood by students, and the odd academic found themselves in difficulty also. This could be attributed to information literacy deficiencies, but there are few cues on LEARN for the uninitiated or confused. A number of other findings arose from the usability testing exercise. LEARN usability testing also dramatically confirmed the dictum that most web users scan rather than read. 12 Links that could answer specific questions (e.g. library hours) were identified more convincingly than those that answered open questions (find information on...). Some people looked to Voyager (Library Catalogue) for solutions rather than LEARN. Perhaps new students are taught that Voyager is where to begin your quest for information; they associate the Library with the catalogue and may not be accustomed to looking beyond it. The Voyager homepage duplicates some LEARN links, so there is a blurring of which resource (i.e. LEARN or Voyager) does what. Again, people may not understand the kind of information Voyager provides. Two of the tasks were poorly worded and resulted in low-level performances: Q7 You're doing research or a project from home and need some help. How would you get it? 12 Jakob Nielsen, Designing Web Usability (Indianapolis: New Riders, 2000). pp Page 8 of 19

9 Q3 You need to find some information on genetic engineering. Where would you start? Q7 was attempting to test the Ask-a-Librarian link. Some participants thought the question was referring to hardware or network assistance; others wanted to contact Library staff directly. This scenario could have been made clearer. Q3 was attempting to test the Resources by Subject link, yet many participants used Voyager to answer this question instead, which isn't wrong. Voyager should have been ruled out. Better design and pretesting would have alleviated this; language difficulties were another factor. The use of small groups as advocated by Neilsen was validated in our testing. The two small groups (one of twelve; one of eleven) showed significant commonality for top level issues. Although the groups were somewhat heterogeneous, the staff/student distinction which might have been a logical divider was blurred in terms of both IT and, to a lesser extent, information literacy competencies. Search behaviours exhibited on secondary pages, as users tried to complete tasks, were often very revealing and challenged library staff s assumptions. There is additional food for thought contained in the transcript data which could be usefully applied to these pages at some point. It was tempting to draw conclusions as for statistically significant test procedures. However the LAG team tried not to act on anomalous behaviours or data, but to interpret the usability testing results in terms of a degree of commonality between participant responses. Other review processes stemming from the experience and expertise of information professionals remained significant. Usability testing data and insights were therefore contextualised; not taken as gospel. Nevertheless, the exercise provided much-needed user input into the redesign of the LEARN homepage. The second testing series reinforced the notion that LEARN is an evolving entity; more could be done to elucidate gateway links. If initial choices were inappropriate, some users were easily discouraged. Conclusion It was a fascinating exercise, watching people navigate the library homepage. Expectations of user behaviour were profoundly challenged as academic researchers struggled to find information, and students approached the site with strategies that seemed sensible to them, but didn t fit the library paradigm or existing configuration of the pages. Other researchers concur: Discovering users search behaviours and their needs was a transforming experience, especially for those librarians who actually administered the test Far from being downhearted by the low rates of success that the test revealed, we were empowered and energized by knowing what needed to be addressed. 13 Usability testing helped identify areas which could be addressed by changes to the homepage, and a few alterations to key secondary pages were made too. What also became obvious was that user difficulties often related not so much to navigation, but to conceptions of how library online resources are organised and function. The transcripts are a goldmine in this regard, and further analysis could provide substantial insights into users search behaviour and expectations. (See Appendix 4) 13 Barbara J. Cockrell and Elaine Anderson Jayne, "How Do I Find an Article? Insights from a Web Usability Study," The Journal of Academic Librarianship 28, no. 3 (2002). p. 131 Page 9 of 19

10 The evaluation of LEARN using this form of testing was something of a pilot exercise. Usability testing was an untried tool in the Library, and has not been universally acclaimed. Vaughn and Callicott expressed disappointment at its limitations, and asserted that many user difficulties stemmed from unfamiliarity with libraries and underdeveloped research strategies. They caution against confusing ease of use with usefulness, and suggest that redesigning library web pages on the basis of usability testing results alone may compromise necessary complexities. Their assertion that library web pages are qualitatively different from commercial sites frames these conclusions, but begs the question as to whether users would change scanning behaviour, for example, just because they are on a library website. Nevertheless, Vaughn et al acknowledge benefits from usability testing. 14 Norlin and Winter s guide to usability testing of library web sites acknowledges its shortcomings in terms of inadequate representation, the test environment and possible skewing of results, but concludes that the methodology s strength in obtaining direct user data well justifies its use. 15 Test facilitators also discovered that usability testing was very rewarding, and learnt a great deal about the method, our users and our own assumptions. Putting it together took time and effort, but not as much as might be anticipated and it has been worth it. Through a combination of organisational, positional and wording changes, the LEARN of 2004 has become the LEARN of today. The Library homepage has improved in usability, and its redesign has been directly influenced by user feedback. The repeat testing validated many of the changes made to LEARN, with a marked increase in user success rates on certain tasks. Of course web pages continually evolve, and it is acknowledged that some modifications didn't work as expected and require further attention. As with all things related to developing and maintaining websites, this is an ongoing process. Usability testing added impetus, direction and validation to the 2004 review of LEARN. The information architecture of LEARN can materially help or hinder University Library patrons and it is essential to incorporate data gained directly from their behaviour and experience into its design. Further analysis could feed into much of what the Library produces: in print or online; in teaching and learning; in how we organise library information whether bibliographic, promotional, instructional or functional. Most of all it could help library staff understand our users; how they think and behave; how we can assist them more effectively. 14 Debbie Vaughn and Burton Callicott, "Broccoli Librarianship and Google-Bred Patrons, or What's Wrong with Usability Testing?," College & Undergraduate Libraries 10, no. 2 (2003). pp. 15, Elaina Norlin and CM! Winters, Usability Testing for Library Web Sites: A Hands-on Guide, A.L.A. Editions (Chicago: American Library Association, 2002). pp. 5, 6 Page 10 of 19

11 Bibliography Battleson, Brenda, Austin Booth, and Jane Weintrop. "Usability Testing of an Academic Library Web Site: A Case Study." The Journal of Academic Librarianship 27, no. 3 (2001): Cockrell, Barbara J., and Elaine Anderson Jayne. "How Do I Find an Article? Insights from a Web Usability Study." The Journal of Academic Librarianship 28, no. 3 (2002): Davidsen, Susanna, and Everyl Yankee. Web Site Design with the Patron in Mind : A Step-by- Step Guide for Libraries. Chicago: American Library Association, Krueger, Janice, Ron L. Ray, and Lorrie Knight. "Applying Web Usability Techniques to Assess Student Awareness of Library Web Resources." The Journal of Academic Librarianship 30, no. 4 (2004): Nielsen, Jakob. Designing Web Usability. Indianapolis: New Riders, Why You Only Need to Test with 5 Users, 2000 [cited 24th August 2004]. Available from Norlin, Elaina, and CM! Winters. Usability Testing for Library Web Sites: A Hands-on Guide, A.L.A. Editions. Chicago: American Library Association, Rubin, Jeffrey. Handbook of Usability Testing : How to Plan, Design, and Conduct Effective Tests. New York: John Wiley & Sons, Inc, Vaughn, Debbie, and Burton Callicott. "Broccoli Librarianship and Google-Bred Patrons, or What's Wrong with Usability Testing?" College & Undergraduate Libraries 10, no. 2 (2003): Veldof, Jerily R. "Usability Testing." In Developing Web-Based Instruction : Planning, Designing, Managing and Evaluating for Results, edited by Elizabeth A. Dupuis, New York: Neal-Schuman Publishers, Further Readings Allen, Maryellen. "A Case Study of the Usability Testing of the University of South Florida's Virtual Library Interface Design." Online Information Review 26, no. 1 (2002): An, Jae Soon, Su-Hong Park, and Eun-Ok Baek. "Web Usability Test Findings and Analysis Issues." Paper presented at the Proceedings of selected research and development [and] practice papers presented at the National Convention of the Association for Educational Communications and Technology, Atlanta, GA, November 8-12, Cobus, Laura, Valeda Frances Dent, and Anita Ondrusek. "How Twenty-Eight Users Helped Redesign an Academic Library Web Site." Reference & User Services Quarterly 44, no. 3 (2005): Dent, Valeda Frances. "Innovation on a Shoestring : An All-Virtual Model for Self-Paced Library Orientation on an Urban Campus." College & Undergraduate Libraries 10, no. 2 (2003): Jasek, Chris. How to Design Library Web Sites to Maximise Usability Library Connect, 2004 [cited 27 August 2004]. Available from Krug, Steve. Don't Make Me Think! : A Common Sense Approach to Web Usability. Indianapolis: New Riders Publishing, Page 11 of 19

12 McKnight, Cliff, Charles Oppenheim, and S.M. Zabed Ahmed. "A Study of Users' Performance and Satisfaction with the Web of Science Ir Interface." Journal of Information Science 30, no. 5 (2004): Nielsen, Jakob. Use-It.Com: Jakob Nielsen's Website, 2005 [cited 5th August 2005]. Available from University of Washington Libraries. Usability, 2005 [cited 28 August. Available from Page 12 of 19

13 List of Appendices Appendix 1 Facilitator Briefing...14 Appendix 2 Questions used to test LEARN...15 Appendix 3 Results Summary...16 Appendix 4 Transcript Excerpts...17 Appendix 5 LEARN Appendix 6 LEARN Page 13 of 19

14 Appendix 1 Facilitator briefing Objectives To incorporate user feedback into web design process To inform design of LEARN front page i.e. primary target is use of 1st level links and menus Usability Testing Qualitative method drawing inferences from close observation, rather than applying statistical methods to questionnaire responses, or counting hits Based on premise that people don t necessarily do what they say they do Useful inferences can be drawn from observing how a site is used Particularly useful and suitable for evaluating web sites, and has been used extensively over last decade Approach participants Explain wanting feedback on library web pages don t mention LEARN by name Not a test ; all experiences helpful About an hour at a mutually convenient time and place Your name will not be recorded in any results. Structure Intro put at ease Few easy questions eg what subject area, how do you link to library web pages. Task oriented section observe and transcribe Discussion section questions and opportunity to comment on tasks Other comments Page 14 of 19

15 Appendix 2 Questions used to test LEARN General Questions 1. Which of these groups best describes your status at University? 2. What is your main subject area, department, or administrative unit? 3. How long have you been at this University? 4. How often do you use the Library web pages? 5. Have you ever heard of LEARN? What is it? 6. Tell me how you would get to LEARN. Task Questions 1. You want to find out if the Library has The Lord of the Rings. Where do you go to find it? 2. You have been asked to find journal articles on global warming. How would you find them? 3. You need to find some information on genetic engineering. Where would you start? 4. A book you want on Voyager is in the Tamaki Library. How would you get it? 5. You want to know how many books you are allowed to borrow at a time. Where would you find this information? 6. You want to know when the Library closes on Saturday. How would you find out? 7. You're doing research or a project from home and need some help. How would you get it? 8. You've heard that the Library has some web pages with information about how to set out references or a bibliography. Can you find them? 9. You've heard that the Library runs courses that can be booked online. How would you book a course? 10. You'd like to find the Subject Librarian for your subject. How would you find them? Discussion Questions 1. What do you like about LEARN? 2. What don't you like about it? 3. What would you like to have on LEARN that isn't already there? 4. What do you think would make it easier to use? 5. Have you ever been unable to find something on LEARN that you expected to be there? 6. Other comment (take notes : 5 mins) Page 15 of 19

16 Appendix 3 Results summary Page 16 of 19

17 Appendix 4 Transcript excerpts The following transcripts illustrate the difficulty participants had with the open ended questions, e.g. Q 7: You're doing research or a project from home and need some help. How would you get it? As opposed to the closed questions that participants found easier e.g. Q 6: You want to know when the Library closes on Saturday. How would you find out? "RC explains what question means: you need help at home when using an online resource on LEARN. I would use Google clicks on Library Contacts link. Does this mean staff? Clicks Voyager link and selects the guided keyword search. This is how I would get information. RC: What about if you were halfway through a search and got stuck? If I get stuck? I'm not sure. Some help from the Library? I'd need to access the web. Get onto a page. See what kind of help I need to get I would never call the Library!! Wait until Monday " The following transcripts illustrate the conceptual problems participants had, with things like articles, serials, how Voyager works etc, e.g. Q 2: You have been asked to find journal articles on global warming. How would you find them? " Maybe to go Voyager. Goes to Guided Keyword and searches for Global Warming. Question is read again. Oh, journal articles! Still in Guided Keyword. Changes 'Search in' to 'Title'. Can't search it here? Goes out to LEARN and clicks e-journals. Types 'global warming' as a title. No hits. Have I finished? Clicks on e-journals. Searchs for 'global warming' as a keyword in title. Can't find anything. ' E-journals closest thing Types 'global warming' as a keyword in title. Failed. Goes to Voyager home page. Nothing says journals to me except UoA e-journals No results. Back to LEARN. Uses LEARN search engine and enters 'journals'. Uses [resulting] link to Voyager main search list. Selects serial title. I didn't match 'serial' with 'journal'. Types in 'global warming' limited to serials. I'm pretty confused and I don't know why Should I search in Voyager or go back to the main page? [Searches by title in Voyager]. Will this give me a complete book or article? Looks at the results. Well. Not clear. Goes to Databases and Article Searching. Looks at page. I'm gathering that it would be under science. Clicks Science then Geography. Looks at links. Are they links to journal articles? Clicks Science Direct and finds some articles. How do you find out what the University has? Is that e-journals? Clicks on Databases and Article Searching. Clicks H and selects Hein Online database (law). RC explains that that is a law database. That is my subject. I use this before. Is this not right? Clicks back and hovers mouse over G (list of databases starting with G). I don't really understand the question. [Looking for information on 'genetic engineering'] Goes to homepage. Clicks on Voyager, Main Search List. Does a title search for genetic engineering. Finds 56 hits. " Page 17 of 19

18 Appendix 5 LEARN Page 18 of 19

19 Appendix 6 LEARN 2005 Page 19 of 19