QUT Digital Repository:

Transcription

1 QUT Digital Repository: Shaw, Kate and Spink, Amanda H. (2009) University library virtual reference services : best practices and continuous improvement. Australian Academic & Research Libraries, 40(3). pp Copyright 2009 Australian Library and Information Association Ltd and Authors

2 UNIVERSITY LIBRARY VIRTUAL REFERENCE SERVICES: BEST PRACTICES AND CONTINUOUS IMPROVEMENT Kate Shaw, Amanda Spink The inclusion or not of chat services within Virtual Reference (VR) is an important topic for university libraries. Increasingly, supported by a Frequently Asked Questions (FAQ) database is suggested in the scholarly literature as the preferred, cost-effective means for providing university VR services. This paper examines these issues and identifies some best practices for university library VR services relating to chat and service, collaborative service provision, services staffing, and staff training. Further studies are required to more completely identify best practices for the complete range of VR services. AARL September 2009 vol 40 no 3 pp Kate Shaw, Queensland University of Technology Library, Gardens Point Campus, Brisbane, Qld, [email protected] Amanda Spink, Research Capacity Building Professor of Information Science, Queensland University of Technology, Gardens Point Campus, Brisbane, Qld, [email protected] INTRODUCTION This paper discusses the concepts of Virtual Reference (VR) and best practice, outlines VR service tools and service provision, and examines the evaluation of VR services. It provides a definition of best practice in and chat service provision, compares independent and collaborative VR approaches, discusses staffing and staff training, and analyses methods for evaluating and continuously improving VR services. Much of the international scholarly literature on university library VR best practice analysed within this paper has been published in the last eight years. Of the papers examined, twenty three were identified as empirical papers, and sixteen were opinion pieces. 192 Volume 40 Number 3 Australian Academic & Research Libraries

3 University library virtual reference services: best practices and continuous improvement Virtual Reference services and best practice Virtual Reference has been defined as a reference service initiated electronically often in real-time, where patrons employ computers or other Internet technology to communicate with reference staff, without being physically present. 1 Virtual reference is also called digital reference, e-reference, electronic reference, remote reference, Internet information services, live reference, and real-time reference. 2 This paper focuses on and online chat services and instant messaging as the predominant VR service tools used and thus the most discussed in the literature. Morin suggests that best practices and guidelines outline a process, practice, or method that can improve effectiveness and efficiency in several situations. 3 Although best practice is not easy to define, Morin believes library best practices should be task specific. 4 Best practices and guidelines provide opportunities to define quality services and introduce new assessment measures, assist with task delineation, save staff time by outlining scenarios and options, set expectation levels clearly, smooth staffing transitions, and identify gaps in understanding. 5 Lankes suggests that establishing best practice within VR services is advantageous to fostering better linkage between research and practice. 6 Lankes also raises the concern that money is being invested in services, software, and planning without benefit of clear empirical research pointing to best practices and without benefit of clear understanding of the advantages and shortcomings of these reference services. 7 Morin suggests that VR service best practice can be used as a guide to ensure consistency during the implementation of a VR service, and goes on to recommend that best practice and guidelines can be developed by: reviewing digital reference guidelines for comparable reference services; assessing existing reference policies and guidelines to identify what is being emphasised; examining current practices by checking with librarian s tacit knowledge; comparing the organisation s mission and goals to patron s needs, and utilising any new software or recent organisational changes to improve processes. 8 Kresh supports the need to establish standards based on best practice when developing a VR service. She also emphasises that the service s vision or mission, staffing needs, and patron needs should be used to assess what is best for it to be effective. 9 Strong proposes there are likely no formula answers to delivering quality VR services and encourages people to think creatively about solutions in our virtual space. 10 Strong also suggests that an evolving digital reference services model should be adopted. 11 Additionally, Wasik provides a structured six step process that organisations should follow in order to implement any successful VR service: informing, where preliminary research into areas of expertise and existing service areas is conducted; Australian Academic & Research Libraries September

4 Kate Shaw, Amanda Spink planning, where procedures, methods, and policies that reflect the overall organisational goals are developed; training, including developing a training plan to prepare staff for the service; prototyping, where the service is tested and modified before launch; contributing, which involves publicity and resource development for service support; and evaluating, which includes regular service evaluation to identify improvement opportunities. 12 Prior to establishing a chat service, Radford and Kern advocate conducting a user needs assessment to determine the optimal hours and days of a service. They also recommend evaluating software needs, vendor options, institutional technology needs, and suggest speaking to similar institutions that have already introduced a VR service. They go on to suggest that it is not unreasonable to allow a minimum of two years to pilot or establish a new virtual chat service before determining its success or otherwise. 13 Currently, further research is required in order to determine the most effective tools a VR service should provide and how a VR service should operate in order to achieve best practice. Research is needed to determine the types of questions best suited to different media and the questions that can be adequately answered without human intervention. 14 Gross, McClure, and Lankes state that the literature indicates that reference works best for ready reference or simple factual questions that require few resources and can be stated in an unambiguous way. 15 However, they note that Short Messaging Services (SMS), video conferencing, chat, and other applications from the commercial sector still require evaluation from libraries in order to determine their usefulness across the range of user questions. They go on to propose that new VR service models are needed for state of the art reference service and improvement in quality across the range of VR services. 16 VIRTUAL REFERENCE SERVICE TOOLS and chat Controversy exists within the literature regarding the viability of chat services in cost/benefit terms. Increasingly, supported by a Frequently Asked Questions (FAQ) database is suggested as a preferred and more cost effective means of VR service provision. There is also discussion regarding the number of support hours a VR service should provide 17 as patrons grow to expect that and chat VR services will be available over extended hours of operation. 18 The inclusion of a chat facility within the VR service has been proposed by Zanin-Yost who suggests chat and patrons should: be asked questions to clarify the librarian s understanding of the patrons needs; 194 Volume 40 Number 3 Australian Academic & Research Libraries

5 University library virtual reference services: best practices and continuous improvement be provided with a variety of accessible information; be asked if their question was answered; be referred to a library subject specialist if required, and for chat patrons, be sent messages which summarise the librarian s activities. 19 Ciccone and VanScoy suggest patrons should be sent scripted please hold messages and instant messaging should be utilised to ask other librarians for back up assistance during busy chat periods. 20 Barbier and Ward suggest that the best chat of the month chats could be used to highlight good chat techniques and to monitor quality. They suggest that chat transcripts that indicate software difficulties, difficulty with chat techniques, or rudeness should be referred to the service coordinator who can assist reference staff to handle such problems. 21 Coffman and Arret note that there are ways to reduce the cost of chat reference services and that alternative services may be preferred. Chat reference service costs may be reduced by staffing the service from the regular reference desk, contracting out staff, joining consortia, or by using free or low-cost systems like AOL, MSN, or Yahoo! Instant Messaging to provide the service. 22 Similarly, Radford and Kern suggest using free instant messaging software to reduce chat service cost. 23 Coffman and Arret point out that telephone reference services, better services and self-service capabilities exist as alternatives to a chat reference service. 24 In contrast, Radford and Kern recommend providing telephone, , and chat within the VR service, and encouraging overlap and referrals between the range of VR service tools. 25 An FAQ database has been proposed as a way of improving VR services. Allowing patrons to use self service can reduce staff numbers and hours required for a chat or VR service, and reduce the number of repetitive or straightforward operational questions handled by staff. Pomerantz, Nicholson, Belanger, and Lankes studied the paths VR services take in a general process model of asynchronous VR. They found that 35% of respondents wanted the FAQ to be automatically searched when a question was received and 15% of respondents wanted access to stored FAQs. 26 Although, Ciccone and VanScoy state that developing a knowledge base of reference answers is useful for answering questions and making referrals, they suggest that it is not useful if staff are primarily conducting reference interviews and instructing patrons. While Gross, McClure, and Lankes suggest that providing canned question responses and developing FAQ pages may meet user needs in specific situations, these approaches need to be assessed for their overall effectiveness and positive impact on the service more generally. 27 VIRTUAL REFERENCE SERVICE PROVISION The literature identifies a range of staff attributes, service levels, and training that are necessary to provide a successful VR service which may be delivered as a stand-alone operation or in cooperation with other agencies. Australian Academic & Research Libraries September

6 Kate Shaw, Amanda Spink Independent versus collaborative service provision Virtual reference can be provided as an independent operation or as a collaborative venture with others. A collaborative VR service comprises an online network of libraries using their cumulative local knowledge and collections to provide VR services to patrons from any of their members. 28 Independent chat service provision refers to the provision of a service within the one location. Radford and Kern s case study of six independent and three collaborative chat reference services identified that the major reasons for chat services discontinuation are funding problems and low volumes of patronage. 29 Bailey- Hainer argues that collaborative chat services provide increased chat service hours for a lower cost and are thus more likely to remain viable. 30 However, Gross, McClure, and Lankes report that collaborative chat services present their own cost issues. One example is the matter of how the cost of developing, managing, and providing a collaborative chat service is to be shared amongst partners. 31 In terms of what is required to provide a successful collaborative chat service, Radford and Kern suggest a full time coordinator, a variety of libraries, and a strong commitment from all libraries within the service to be necessary for successful service provision. They go on to report that libraries within a collaborative service need to encourage ongoing communication and support amongst themselves; have aligned institutional missions, staffing patterns, and user groups; and share a collaborative chat service vision. 32 Staffing and staff training In any service, Strong suggests that staff should be proactive, knowledgeable, well-trained active listeners, able to respond to all inquires, neutral in their opinions about the information provided, or, in cases where personal opinions are provided, willing to acknowledge them as personal opinions. 33 Stormont, Boyer, and Francoeur suggest that staff demonstrate keyboarding skills, a multi-tasking ability, good searching skills, an ability to write concise messages, and an ability to deal with stress and demanding users. 34 Radford recommends staff also possess interpersonal skills, such as techniques for rapport building, compensation for lack of nonverbal cues, strategies for relationship development, evidence of deference and respect, face-saving tactics, and greeting and closing rituals. 35 Virtual Reference staff service models have been proposed. Chat is often performed away from the physical reference desk with additional staff hired for service provision due to the demands of chat services. 36 McClennen and Memmot suggest the VR service be staffed using a specified role model for information professionals, including filterer, answerer, administrator, and coordinator. Different staff being employed for each of the four roles: the filterer responding to repeated and trivial questions; an answerer concentrating on more complicated questions; the administrator ensuring smooth service operation, adding answers to the FAQ database, creating user accounts, and performing other technical tasks and 196 Volume 40 Number 3 Australian Academic & Research Libraries

7 University library virtual reference services: best practices and continuous improvement a coordinator defining and implementing policies and procedures, and initiating service improvement. 37 Similarly, Strong proposes a triage model for VR service staffing. Virtual reference questions being grouped into three broad categories, directional and inquiry; strategy and tutorial; and research assistance. Directional and inquiry questions are answered by library assistants at the first level, strategy and tutorial question are answered by second-level librarians, while questions that require subject expertise are answered by third-level subject specialists. 38 All of these approaches require considerable management, staffing resources, and administrative effort in order to ensure they can be implemented in a practical and cost effective manner. VR service staff training recommendations have also been described. Cheat sheets, online tutorials, the provision of an on call person to assist with technical difficulties, flexible exercise completion times, and a variety of training techniques to cater for different learning styles have been identified as effective approaches for conducting online VR training. 39 Traditional presentations, hands-on practice, written training documentation, and one-on-one training sessions have also been recommended. 40 Ciccone and VanScoy suggest practicing responses to VR queries, discussing strategies for dealing with multiple patrons at once, and considering ways of handling problem patrons. 41 Furthermore, provision of centralised and similar training levels for all VR service staff 54 and staff training that includes train the trainer sessions, ongoing training opportunities, transcript review, and high levels of staff communication have been recommended. 42 EVALUATING VIRTUAL REFERENCE SERVICES To identify VR service best practice requires service evaluation. Findings in VR service tools, service provision, staffing, and staff training highlight some best practice recommendations. Service evaluation methods include techniques borrowed from traditional reference, evaluation guidelines, new approaches, and assessment to move beyond traditional conceptions of reference services. 43 Gross, McClure, and Lankes propose Murfin and Bunge s four methods for assessing cost effectiveness in academic libraries as a starting point for VR service evaluation focusing on costs. 44 These methods include: 1. measuring the full cost of reference transactions; 2. using a reference service cost effectiveness index based on success, helpfulness, accessibility, and time or cost; 3. determining the cost or time taken for each successful question, and 4. developing a cost-benefit formula. 45 Other traditional reference service measures include the input/output model that measures inputs and service utilisation in small libraries, and functional cost analysis to define and allocate the costs of service provision. 46 Australian Academic & Research Libraries September

8 Kate Shaw, Amanda Spink In addition, Morin, Nilsen, and Shachaf and Horowitz propose using the International Federation of Library Associations (IFLA) digital reference guidelines, and the Reference and User Services Association (RUSA) guidelines for implementing and maintaining VR services in order to assess service effectiveness. 47 IFLA digital reference guidelines cover service administration and the practice of VR, and are designed to promote international best practice. 48 These guidelines include general principles for the practice of VR, content guidelines for the service, chat guidelines, and specific guidelines for conducting chat sessions. RUSA guidelines for implementing and maintaining VR services are also designed to assist libraries and consortia to establish VR operations. The RUSA guidelines advise on preparing, providing and organising VR services, including privacy issues. 49 According to Morin, although the transition from implementation to a sustainable digital reference service can take years, best practices and guidelines can help ensure consistent service by reflecting practical customs, procedures, and habits during the implementation period. 50 McClure and Lankes propose four types of measurement to understand in order to set VR service benchmarks. These include outcome measures, process measures, economic measures, and user satisfaction: outcome measures are described as the quality of the answers and should be assessed based on response accuracy, response appropriateness to the user audience, the opportunities for interactivity, and the level of instruction provided in the response; process measures as process effectiveness and efficiency, based upon service accessibility, response timeliness, clarity of service procedures, percentage of questions answered, staff training and review methods, and the service s review and evaluation methods; economic measures to assess cost effectiveness, taking into account VR session costs, the infrastructure required to support quality VR services, and the impact on other library expenditures, and user satisfaction, the degree that users of a VR service are satisfied with the process and results, which can be assessed using indicators such as accuracy, timeliness, staff behaviour, technical considerations, and physical facilities. 51 In their consideration of cost effectiveness, Lankes, Gross, and McClure suggest both utilisation and technical standards be considered with costing, statistics, and measuring for VR services. 52 They recommend embedding quality standards and assessment data into software and infrastructure and go on to describe utilisation standards that deal with VR service use, delivery, and success. 53 Technical standards include hard tools use, such as software, hardware, protocols and other standards, as well as soft tools, such as metadata and organisational schema. They propose six quality standards, including courtesy, accuracy of responses, satisfaction level of users interaction with the service, percentage of repeat users, user group s service awareness, and cost per service transaction. They also propose five types of performance success measures for quality standards, 198 Volume 40 Number 3 Australian Academic & Research Libraries

9 University library virtual reference services: best practices and continuous improvement including descriptive statistics and measures to determine service scale and scope, log analysis, user satisfaction measures, ongoing service cost analysis, and staff time expended. 54 Rigby, Smithers and Zhang propose additional methods of VR service evaluation. They outline three key performance indicators (KPIs) to enable evaluation and comparison of operators in New South Wales (Unison) and Council of Australian University Librarians (CAUL) libraries collaborative VR services. KPIs include attributes, analytical skills, support and staffing required to run the service; as well as the knowledge and ability to match resources to user needs. They identify new service standardised measures, including a client satisfaction survey, Rodski survey, service staff training methods and weekly statistical surveys to measure KPIs. 55 In addition, Lou also recommends using a chat reference evaluation framework to evaluate the service and its reception by users (see Appendix 1). He suggests chat reference be evaluated from the service s perspective using statistics, and the content of chat reference transactions and proposes user acceptance be evaluated by measuring their awareness, preferences, status, and feedback perceptions. Lou also recommends using chat reference evaluation manuals to supplement this general framework. 56 Continuous improvement of Virtual Reference services Although significant work has been done in this area, further research is needed to determine the most effective models for service evaluation and continuous improvement. Gross, McClure, and Lankes highlight that most VR literature is anecdotal or editorial in nature, includes weak methods, and provides limited analysis of services. They suggest that case studies often do not provide measures or assess services, propose that evaluation should be approached from a conceptual point of view, and indicate that few people are actually evaluating services. They also found that the scholarly literature includes few economic models, ways to account for VR services, or participation from users in service design, development, and implementation. In addition, they note that VR services need to do better in assigning costs and assessing the most efficient way of providing services. For collaborative approaches, they stress the importance of an awareness of how costs are shared in setting up the service, and emphasise the importance of using input from users to improve service delivery and assess acceptable turnaround times and the design of VR service marketing. 57 Radford found that most studies focus on question accuracy and efficiency. She proposes that more research is needed to understand and improve chat encounter quality and that more empirical studies are needed to address unanswered or partially answered research questions. Future research should involve online surveys, face to face interviews, and focus groups with librarians, clients, and nonusers to provide more definitive answers about the relationship between content and the quality of responses. 58 Australian Academic & Research Libraries September

10 Kate Shaw, Amanda Spink DISCUSSION While some best practices have been identified within the scholarly literature for chat and service staffing, staff training, and independent and collaborative provision of services, further evaluation of services is needed to identify a clearer picture of VR service best practice. Debate exists within the literature about the use of chat and whether chat services are viable in cost benefit terms. Although other suggestions exist for reducing chat service cost and improving chat service effectiveness, increasingly, supported by an FAQ database is seen as the preferred and most cost effective means for providing this service. This approach can be argued to be more cost effective than providing a chat service because patrons are able to self-service, and, generally, less time is spent responding to an inquiry in comparison to time spent responding to chat inquiries. supported by an FAQ database encourages patrons to search for information within the database before contacting library staff. More time is generally spent answering a chat enquiry in comparison to an enquiry because chat enquiries are often less clearly defined and more interactive, resulting in a prolonged encounter. It is also likely that chat enquiries often require time to be spent communicating back and forth with patrons in order to clarify the patron s question before a comprehensive answer can be provided. Further research is required in order to determine whether or not chat services are an effective and economical option for resource constrained libraries. University libraries interested in providing chat services should consider utilising free chat service providers as one option to help reduce their VR service costs. RECOMMENDATIONS Based on this review, further VR service evaluation needs to be undertaken in order to better understand the most effective approaches to adopt. Specifically, online chat service evaluation should be undertaken in order to determine the cost effectiveness of such services. This review of the literature also suggests an service supported by a Frequently Asked Questions database is seen as the most cost effective approach to VR service. Using the IFLA and RUSA guidelines to is a useful approach to adopt in order to evaluate service provision. While traditional reference evaluation methods, the use of guidelines and development of new evaluation methods are suggested as tools for assessing service effectiveness, these methods have been criticised. Further research into appropriate methods for evaluating VR services is recommended. 200 Volume 40 Number 3 Australian Academic & Research Libraries

11 University library virtual reference services: best practices and continuous improvement NOTES 1. American Library Association Reference and User Services Association, Guidelines for Implementing and Maintaining Virtual Reference Services 2004 at layout_1_512081_ cfm viewed 21 August K Nilsen & C Ross Evaluating virtual reference from the users perspective Reference Librarian vol 46 no 95/ pp53-79 at com.ezp01.library.qut.edu.au/store/e-text/view_etext.asp?sid=wp45g TWEF8TR8PDW9EPNECWH0QG3D146&a=3&s=J120&v=46&i=95% 2F96&fn=J120v46n95%5F05 viewed 13 August A C Morin Chapter 9: Approaching Best Practices and Guidelines for Digital Reference R Lankes, J Janes, L C Smith & C M Finneran (eds) The Virtual Reference Experience: Integrating Theory into Practice London Neal-Schuman Publishers Inc Morin ibid. 5. Morin loc cit. 6. R D Lankes Digital Reference Research: Fusing Research and Practice Reference & User Services Quarterly vol 44 no pp Lankes ibid., p Morin loc cit. 9. D Kresh Virtually Yours: Thoughts on Where We Have Been and Where We Are Going with Virtual Reference Services in Libraries B Katz (ed) Digital Reference Services USA The Haworth Information Press 2002/ G E Strong If we change it will they come? Reference Services Review vol 24 no p Strong ibid., pp J M Wasik Building and maintaining digital reference services Eric Digest 2000 at viewed 6 November M L Radford & K M Kern A multiple-case study investigation of the discontinuation of nine chat reference services Library and Information Science Research vol 28 no pp M Gross, C R McClure & R D Lankes Assessing Quality in Digital Reference Services: An Overview of the Key Literature on Digital Reference R D Lankes, C R McClure, M Gross & J Pomerantz (eds) Implementing Digital Reference Services: setting standards and making it real London Facet Publishing Gross, McClure & Lankes op cit., p Gross, McClure & Lankes loc cit. 17. Radford & Kern loc cit. 18. M Ruppel & J C Fagan Instant messaging reference: Users evaluation of library chat Reference Services Review vol 30 no pp Australian Academic & Research Libraries September

12 Kate Shaw, Amanda Spink 19. A Zanin-Yost Digital Reference: What the past has taught us and what the future will hold Library Philosophy and Practice vol 7 no pp K Ciccone & A VanScoy Managing an established virtual reference service Internet Reference Services Quarterly vol 8 no 1/ pp P Barbier & J Ward Ensuring quality in a Virtual Reference environment Community and Junior College Libraries vol 13 no pp Coffman & Arret To chat or not chat: Taking yet another look at virtual reference loc cit; Tenopir Chat s positive side loc cit. 23. Radford & Kern loc cit. 24. Coffman & Arret To chat or not chat: Taking yet another look at virtual reference loc cit. 25. Radford & Kern loc cit. 26. Coffman & Arret To chat or not chat: Taking yet another look at virtual reference loc cit; G Chowdhury & S Margariti Digital reference services: a snapshot of the current practices in Scottish libraries Library Review vol 53 no pp M Gross, C R McClure & R D Lankes loc cit. 28. Ciccone & VanScoy loc cit. 29. Radford & Kern loc cit. 30. B Bailey-Hainer Virtual Reference: Alive and well Library Journal vol 130 no pp Gross, McClure & Lankes loc cit. 32. Radford & Kern loc cit. 33. Strong loc cit. 34. S Stormont Going where the users are: Live digital reference Information technology and libraries vol 20 no pp ; J Boyer Virtual reference at North Carolina State: The first one hundred days Information technology and libraries vol 20 no pp ; S Francoeur An analytical survey of chat reference services Reference Services Review vol 29 no pp M L Radford Encountering Virtual Users: A Qualitative Investigation of Interpersonal Communication in Chat Reference Journal of the American Society for Information Science and Technology vol 57 no pp Boyer loc cit; S Coffman We ll take it from here: further developments we d like to see in virtual reference software Information Technology and Libraries vol 20 no pp ; M Marsteller & P Neuhaus The chat reference experience at Carnegie Mellon University 2001 at andrew.cmu.edu/~matthewm/ala_2001_chat.html viewed 5 November 2008; Stormont loc cit. 37. M McClennen & P Memmot Roles in digital reference Information Technology and Libraries vol 20 no pp Strong loc cit. 202 Volume 40 Number 3 Australian Academic & Research Libraries

13 University library virtual reference services: best practices and continuous improvement 39. E G Abels, P B Kantor & T Saracevic Studying the cost and value of library and information science: Applying functional cost analysis to the library in transition Journal of the American Society for Information Science vol 47 no pp A Kawakami & P Swartz Digital reference: training and assessment for service improvement Reference Services Review vol 31 no pp Ciccone & VanScoy loc cit. 42. T A Stewart Mystified by training? Here are some clues Fortune vol 143 no p184; Kawakami & Swartz loc cit. 43. Radford & Kern loc cit. 44. A Kasowitz, B A Bennett & R D Lankes Quality standards for digital reference consortia Reference and User Services Quarterly vol 39 no pp ; P Shachaf & S Horowitz Virtual reference service evaluation: Adherence to RUSA behavioural guidelines and IFLA digital reference guidelines Library and Information Science Research vol pp ; T A Peters Current opportunities for the effective meta-assessment of online reference services Library Trends vol 49 no pp ; Morin loc cit, K Nilsen Comparing users perspectives of in-person and virtual reference New Library World vol 107 no 3/ pp91-104; T Saracevic & L Covi Challenges for digital library evaluation N K Roderer & D H Kraft (eds) Proceedings of the 37th American Society for Information Science Annual Meeting Washington D.C American Society for Information Science Gross, McClure & Lankes loc cit. 46. M Murfin & C Bunge A cost effectiveness formula for reference service in academic libraries Washington DC Council of Library Resources Morin loc cit; Nilsen loc cit; Shachaf & Horowitz loc cit. 48. International Federation of Library Associations Reference and Information Services Section, IFLA Digital Reference Guidelines 2005 at VII/s36/pubs/drg03.htm viewed 21 August American Library Association loc cit. 50. Morin op cit., p C R McClure & R D Lankes Assessing quality in digital reference services: A research prospectus 2001 at viewed 5 November R D Lankes, M Gross & C R McClure Cost, statistics, measures, and standards for digital reference services: A preliminary view Library Trends vol 51 no pp Lankes, Gross & McClure ibid. 54. Lankes, Gross & McClure loc cit. 55. A Rigby, A Smithers, & S Zhang Unison Digital Reference Key Performance Indicators Project Report 2006 at digitalreference2006.doc viewed 8 August 2008; Australian Academic & Research Libraries September

14 Kate Shaw, Amanda Spink 56. L Lou Chat reference evaluation: a framework of perspectives and measures Reference Services Review vol 36 no 1 pp Gross, McClure & Lankes loc cit. 58. Radford loc cit. 204 Volume 40 Number 3 Australian Academic & Research Libraries

15 University library virtual reference services: best practices and continuous improvement APPENDIX 1: Chat reference evaluation framework Evaluation perspective Evaluation angle Variable Measurement Service perspective Statistics of chat reference transactions Total number of chat sessions within the period of time that was examined Number of successfully and unsuccessfully connected chat sessions Frequency of chat sessions Chat session length Turns taken in chat session Word number in user s question and librarian s response Referral webpage System logs or manual archiving System logs or manual archiving System logs or manual archiving System logs or manual archiving Analysis of transcripts Analysis of transcripts Systems logs Content of chat reference transactions Type of user s questions Quality of chat reference transactions 1. Quality of the answer to user s question Completeness of the answer Correctness of the answer Appropriateness of referral Delivery of the answer Effectiveness of communication Tones and attitudes exhibited by the librarian 2. Evidence of reference interview 3. Completeness of chat session Analysis of transcripts Analysis of transcripts User perspective Users awareness and preferences User status Users preferences of reference service venues Users awareness of the availability of chat reference service The role in which users ask their questions to chat reference service Survey Survey Survey User perceptions or feedback Users motivation to use chat reference service User perceptions on the interface of chat reference service User satisfaction with chat reference service Users willingness to return User perceptions on the advantages of chat reference service User perception on the disadvantages of chat reference service Users use of information received from chat reference service Survey User studies Survey & analysis of transcripts Survey Survey Survey Follow-up survey or interview Australian Academic & Research Libraries September