An evaluative case study of a real-time online reference service

Transcription

1 An evaluative case study of a real-time online reference service Louise Kloss and Yin Zhang The authors Louise Kloss is at Morley Library, Painesville, Ohio, USA Yin Zhang is at the School of Library and Information Science, Kent State University, Kent, Ohio, USA Keywords Reference services, Information, Libraries, Services Abstract This case study evaluated a real-time online interactive reference service Three data gathering methods were used First, general usage statistics were generated electronically from the service provider Second, patron evaluations were collected electronically Third, questionnaires were mailed to the librarians who provided the service The general usage statistics revealed an overall pattern of growth in the number of participating libraries, the number of patrons, the number of questions and the total time spent answering them in minutes Patron evaluations of the service were positive, most of the patrons questions were answered They all liked the service and intended to use it again The librarians responses were also positive Most of the librarians were able to nd quality Internet Web sites containing the answers to their patrons questions and when they could not, they often found the answers in a print resource This nding suggested that regardless of the setting, print resources still had an important role to play in the provision of reference service Electronic access The Emerald Research Register for this journal is available at wwwemeraldinsightcom/researchregister The current issue and full text archive of this journal is available at wwwemeraldinsightcom/ htm Introduction Real-time reference services are becoming widely available in libraries As a new means of offering a traditional library service, it is essential to evaluate what is currently being offered in order to create effective real-time reference guidelines The purpose of this study was to evaluate the materials, outputs, and outcomes involved in the provision of a real-time online reference service using a multi-method approach In order to ful ll that purpose, data were gathered from both the patrons who used the service and the librarians who provided the service Additionally, the service software collected data that revealed the patrons usage pattern during the initial stage of operation The major goals of this study were: to determine whether or not the service met both the patrons information needs and the patrons expectations; to determine whether or not the service met the librarians expectations; and to determine what areas of the service appeared to be working and what areas of service appeared to need improvement The real-time reference service evaluated in this study is the service provided by the Northeastern Ohio Library Association (NOLA) Regional Library System Since November 2000, this consortium has offered its member libraries the opportunity to participate in the provision of live interactive reference assistance to their patrons in real time via the Internet through AskUsQuestionscom (NOLA, 2001) Reference librarians from a number of participating libraries serve as after-hours operators and provide the online reference service from their homes In order to access the service, patrons must rst, click on the AskUsQuestionscom button which is found in various locations on their libraries Web sites On the next screen, they must enter their patron ID This connects them with an e-reference librarian Chat software allows the two parties to communicate with each other Generally, the patrons type queries, and the librarians type responses while simultaneously searching the Web for answers Then, as soon as the librarian nds the appropriate Web sites, he/she pushes Volume 21 Number pp Emerald Group Publishing Limited ISSN DOI / The authors thank the NOLA Regional Library System for providing system usage statistics and the real-time reference librarians who participated in the study

2 pages from the sites into the patrons browsers This study focused on AskUsQuestionscom s rst several months of operation It is hoped that the results will be of some assistance to those who are interested in planning similar real-time online reference services Literature review Real-time online reference is a relatively new way to offer a traditional service A review of relevant literature reveals that most published articles are largely descriptive in nature in explaining what real-time online reference is, how it works, what types of online reference services exist, and why it is needed (Eichler and Halperin, 2000; Lipow, 1999; Oder, 2001) Early published studies on digital reference largely focus on the use of or Web forms as means of providing reference service to remote patrons (Ables, 1996; Bushallow- Wilber et al, 1996; Fishman, 1998; Garnsey and Powell, 2000; Kasowitz et al, 2000) Additional studies indicate that offering live, or real-time reference service, is a fairly recent addition to the more established digital reference services mentioned earlier One such study was a survey of 150 academic libraries conducted by Janes et al (1999) in which they reported that none of the libraries were providing reference services in realtime Instead, and Web forms were the digital reference methods of choice Their survey appears to con rm both how recently and how swiftly some libraries have developed and implemented real-time reference services Tenopir and Ennis (2002) conducted a longitudinal study of reference services in academic libraries They surveyed the academic members of the Association of Research Libraries four times during the past decade from 1991 to 2000 Their ndings suggest that digital information sources and services were being used at an increasingly accelerated rate in the 1990s and that the Internet and the World Wide Web are changing how reference services are delivered A number of papers have been devoted to the use of the Internet as a resource for answering reference questions Devlin and Burke (1997) conducted a study to determine whether or not the Internet was a reliable and an authoritative source of information They concluded that the Internet provides no comprehensive mechanism for identifying 566 quality information and called for a uni ed service interface to help alleviate this problem However, in another study, Zumalt and Pasicznyuk (1998) used depth, accuracy and durability as three measures to determine the effectiveness of the Internet in answering reference questions They concluded that the Internet could provide answers to a signi cant percentage of actual reference questions and that there was little signi cant difference in the accuracy of information obtained from traditional reference sources and Internet sources Koutnik (1997) conducted a systematic study on using the Internet to answer 104 reference questions He concluded that librarians should be using the World Wide Web to supplement their print reference collections However, Koutnik noted that additional time is needed to weed through the list of Web sites returned as relevant to a query More recent studies have focused on investigating users and librarians in digital reference services For example, Ross and Nilsen s (2000) study provides insight into how Internet sources may have changed reference transactions and affected users experiences with reference services Another example is Janes s (2002) survey, which not only reported reference librarians experiences and attitudes towards digital reference services, but also provided a detailed account of online reference services Sloan s (2003) report about a collaborative online reference service offered by eight academic libraries in Illinois provides rst-hand usage statistics and a list of questions asked by the users of the service Still, evaluations by users and librarians of real-time online reference services are few It is hoped that the results of this study of AskUsQuestionscom, conducted during its initial stage of service, will add to the real-time reference service knowledge base in three areas: usage statistics; patron evaluations; and librarians perceptions Methodology To achieve the goals of this study, the following three types of data were gathered (1) System usage data The service software automatically recorded NOLA real-time reference transactions and then statistics were generated to reveal both the usage patterns of the service and the workload of service librarians

3 (2) Patron evaluations At the end of a realtime reference session, patrons were directed to a service evaluation form These data were gathered within the same timeframe as the system usage data To ensure patron con dentiality, no personal or demographic data were gathered (3) Reference librarian/operator survey A questionnaire (see Appendix) was sent to all participating librarians during the rst 6 months of real-time reference operation The following categories provided the underlying framework for the librarians questionnaire: materials, outputs, and outcomes (Allen, 1995, p 209) Additional questions were included to gather demographic data about the librarians The questionnaire was pre-tested by two reference librarians who had similar backgrounds with the participating librarians, but neither of whom was involved in the research project They reviewed the questionnaire and made suggestions for improvement Finally, the questionnaire was mailed to all seven real-time reference librarians/operators All seven librarians returned the survey within two weeks Results General usage statistics Table I summarizes the monthly totals in each of the following general usage categories: the number of participating libraries, number of patrons, number of questions and the total time in minutes There was an overall pattern of growth in the number of participating libraries, patrons, and questions, as well as in the total time in minutes However, a drop in usage occurred during the month of April This might have been due to spring or Easter break Elementary and secondary school Table I General usage statistics January-May 2001 Month An evaluative case study of a real-time online reference service Participating libraries Patrons Questions Total time (min) January ,176 February ,196 March ,446 April ,360 May ,700 children in Northeastern Ohio Public School Systems would have had at least a one-week break from school sometime during this month The librarian responses reported later appear to reinforce this as a cause for the drop in usage because all the librarians reported providing homework assistance for junior high and high school students, and most of the librarians reported providing homework assistance to elementary students Table II comprises the weekly highs, lows, and averages in the following categories: patrons, questions, and time in minutes NOLA s real-time online reference service was in operation ve days per week and averaged 76 patrons per day The average number of patrons per week and the average number of questions per week imply that most of the AskUsQuestionscom patrons were asking only one question per session Patron evaluations Patron evaluations of the service were very positive, as shown in Table III For 1 month, during the same time period when the usage statistics were being gathered, patron evaluations were also gathered In March 2001, 193 patrons tried the service and 12 of them were randomly directed to the patron evaluation form Of these 12 patrons, only eight responded The response rate was low largely due to the fact that the survey site was pushed at the end of a real-time reference Table II Weekly highest/lowest/average statistics January-May 2001 Highest Lowest Average Patrons Questions Times (min) No data No data 8:13 Table III Patron evaluations of service How helpful was the librarian? Very helpful 6 Helpful 1 Not helpful 0 Was your question answered? Yes 7 No 1 Did you like the service (AskUsQuestions)? Yes 8 No 0 Do you plan to use our service again? Yes 8 No 0 567

4 session In most cases, patrons had logged off before the survey page showed up after the service While the number of respondents was small, it could be considered as a random sample Patrons responses suggest that their experience with the online reference service was a positive one In fact, all patrons indicated that they would use the service again Seven out of eight patrons stated that their question was answered, and six out of seven thought that the librarian who provided the service was very helpful Reference librarian survey results Librarians backgrounds and training for real-time reference The demographic data provided by the reference librarians indicated that the librarians had both the technical skills and practical experience required to conduct the online reference transactions Five out of seven had a Master of Library Science (MLS) or Master of Library and Information Science (MLIS) degree, and the other two had a Bachelor s Degree Six out of seven indicated that they had used the Web to answer reference questions at least once a day during the past year, and all of them had independent reference experience in their libraries They indicated that they had learned how to conduct Web searches and how to evaluate the quality of Web sources mostly through continuing education workshops, self-learning, and in-service training sessions The librarians experiences serving as reference librarians ranged from almost ve to 22 years, and their Web reference experiences ranged from four to six years All librarians received training in all aspects of the software that they would be using for the real-time reference work They received at least a three-hour lecture demonstration training session with the balance of the day for hands-on practice Further, each librarian received an operator s manual When asked to rate how dif cult it is to use the NetAgent software interface, six of the seven librarians rated it as easy to very easy, and one rated it as neutral When asked how well prepared they felt when they logged on for their rst real-time interactive online reference shift, three of the seven felt prepared, three felt neutral, and one felt less prepared Workload and experiences The librarians worked two to 25 hours (one to 125 two-hour shifts) each week providing 568 real-time reference services In the rst four months after the real-time reference service was launched, the librarians experienced three to eight times when they did not have any patron to assist during their shifts Six of the seven librarians found it easy to work alone and commented that they were comfortable providing reference assistance in a nontraditional setting located outside the library walls, for example, from home It is interesting to note how they described their rst real-time reference experience Hard remembering how to use the software Fumbled around a bit, but I think that is normal when using something new A bit nervous, but well motivated and energized Scary Ðbut turned out to be easier than I feared it would Also Ðhad a wonderful co-librarian at my side It was a little scary and awkward to try and do two things at once Ðchat and seriously search the Web Sometimes your mind goes blank as you watch the clock tick away and record the transaction time I d practiced a bit and the workload wasn t heavy, so it wasn t dif cult The only thing dif cult was not knowing what to expect I was familiar with instant messaging and chat, but had not done it too often That was my biggest problem at rst Apparently, even though all the librarians were trained and had the necessary experience for real-time reference work, their rst time using the system still appeared to be challenging The Internet vs traditional print sources as reference sources in real-time online reference Although the majority of librarians speci ed that they were able to meet their patrons information needs using Internet Web sites, their experiences demonstrated that print resources continued to play an important role in the provision of reference services regardless of the setting The majority of the librarians were able to locate quality Web sites containing answers to the patrons questions This result is consistent with the data on patron satisfaction evaluations reported earlier Only one librarian was able to answer all questions

5 using the Internet for reference sources The other six reported that they had at least one time when they could not use Internet sources to answer patron questions In addition, four of the seven librarians reported that they could always nd Web sites that contained authoritative, accurate, reliable and up-to-date information, but only three could locate Web sites that contained comprehensive information about a subject all the time It is interesting to note that although all of the librarians had a collection of pre-selected Web sites for their reference service, they did not rely solely upon their collections to answer patrons questions Instead, they relied upon a combination of approaches that involved searches in their Web reference collection, searches of the Web and occasional searches through print resources At times, some of the librarians consulted either print resources housed within the traditional library or print resources that they had on hand in their homes Sometimes, the librarians requested a patron s address so that they could forward on any information that they later located in their library s print collection Types of questions in online reference transactions Most of the librarians found it dif cult to give the same level of service to two or more patrons simultaneously This implies that before the librarians logged on for their rst real-time reference shift, more time to practice answering more than one question at a time via chat might have been bene cial Table IV summarizes types of questions received during real-time reference work as reported by the real-time reference service librarians It appears that homework questions were most common in real-time reference services All of the librarians reported receiving homework questions for Grades 7-12; ve of them received homework questions for Grade K-6 Six of seven librarians also received questions that were ready reference in nature, or questions that were for recreation/entertainment and personal interest; four of them received questions that were about business or consumer/product information Librarians perceptions of patrons expectations about real-time reference service Most librarians ( ve of seven) felt that their patrons had unrealistic expectations about 569 Table IV Types of questions in real-time online reference transaction Types of questions how long it would take to search for information Also, most librarians indicated that their patrons were unaware of the limitations of Web resources Librarians responses were mixed regarding whether patrons could clearly communicate information they needed in writing Two of seven librarians were on the positive scale, one was neutral, and four were on the negative scale, including one very negative Bene ts and limitations of real-time online reference service and suggestions for improvements Convenience and service after library operation hours were mentioned most often as bene ts of real-time reference Other bene ts mentioned by librarians included the following Library can extend hours in an inexpensive way Can reach a net-based clientele that may never walk into a library Improve the image of libraries Make libraries seem high-tech and with-it Additional educational and training for patrons by helping people nd authoritative and accurate answers and sources Most librarians felt that the hindrances to provide real-time online reference were both Received by number of operators Homework help: Ready reference 6 Recreation/entertainment 6 Personal interest 6 Homework help: K-6 5 Business 4 Consumer/product information 4 Do-it-yourself/how-to-do-it 3 Government 3 Health 3 Other (Ohio information/local history/sports) 3 Hobbies 2 Research assistance: undergraduate 1 Economics 1 Politics 1 Legal 1 Taxes 1 Research assistance: graduate 0 Research assistance: Doctoral 0

6 the amount of time it took to search the Web and the dif culty involved in nding information on the Web This reinforces the results reported earlier regarding the librarians perceptions of the patrons expectations of real-time reference transactions The following are limitations of real-time reference service as mentioned by the participating librarians In depth research We provide breadth of knowledge, not in-depth, generally speaking Only text based chat to communicate Not everything is on the Web Searching the Web can be dif cult Many people seem to think that when they send us a question, we are a search engine and will instantly fetch the correct answer I often have to tell them that I have to keep looking because the information was not at the rst Web site I found, etc Can t always nd info on the Web expediently Reference interviews can be imperfect Communication is limited and sometimes things can get misinterpreted Harder to conduct reference interview Not enough authoritative info on the Web Some questions are dif cult, and it is very time-consuming to locate the exact answer As a result, instant answers aren t always possible, and I need to consult print sources So, having a dial-up access service for the Internet can be very slow and frustrating In terms of improvement in real-time service, most of the librarians felt that voice-over IP would improve the quality of service that they provided to AskUsQuestionscom patrons This reinforces the results reported earlier that most of the librarians reported feeling that both they and their patrons experienced varying levels of dif culty communicating clearly via chat Some suggestions offered by the librarians are as follows De nitely need cable for this 56 K is too slow Voice over IP A good PR and marketing strategy plan will add to the success of an innovative service When most people have high speed connections, voice-over IP communications would be helpful 570 I would like more people to give me their addresses so I could do further research and send them the information Expansion of available hours Voice contact Lists of good search engines and metasearch engines that we can package and send to patrons for their own further searches Conclusions and discussion The overall pattern of growth re ected in the general usage statistics suggests that an increasing number of patrons of the participating Northeastern Ohio libraries began using the service once it was in operation Further, the patron evaluations indicated that patrons were satis ed with the services they received and that they intended to use the service again The results of the real-time reference librarians survey indicated that the librarians felt con dent about their ability in using the Internet both as a resource for answering reference queries and as the means of delivering information However, despite the training and experience each librarian received before his or her rst shift in providing real-time reference service, the rst time still appeared to be challenging In addition, most of the librarians found it dif cult to give the same level of service to two or more patrons simultaneously This implies that before the librarians logged on for their rst real-time reference shifts, more time to practice answering more than one question at a time via chat might have been bene cial On the other hand, it may also suggest that assisting one patron at a time would ensure quality service to all patrons Problems experienced by the librarians during the rst six months in real-time reference service are largely due to the limitations of the service For example, communicating clearly using text based chat appears to be challenging; searching for answers is sometimes more complicated and time-consuming than patrons realize; and patrons may have unrealistic expectations on timely feedback during the reference transaction with a slow network connection As a result, the majority of librarians recommended voice over IP as a possible solution to some of these problems

7 Although the majority of librarians speci ed that they were able to meet their patrons information needs using Internet Web sites, print resources continued to play an important role in the provision of reference services regardless of the setting As a result, this study broadens Koutnick s (1997) conclusion that in the traditional reference setting librarians should use the Internet to supplement their traditional print reference resources Data collected in this study demonstrate that in this non-traditional reference setting, the librarians used traditional print reference resources to supplement their use of the Internet as the primary reference resource As a result, one can conclude that print resources still have an important role to play not only in the traditional library reference setting, but also in the non-traditional virtual reference setting A follow-up study would be necessary to compare how the usage data change over time and how the reference librarians perceptions of this service change as they gain more experience In addition, suggestions for improvement of online reference service can be gathered directly from patrons Furthermore, a follow-up study could address some of the questions that have not been answered in related studies: Are any of the patrons repeat users? How many questions are being answered in addition to real-time reference questions? How many of the total questions asked are unanswered by the librarians? How many of the total number of questions asked are not answered during the session, but are answered via at a later time? What type(s) of questions are most frequently asked? How many are answered using Web sites, and how many are answered with print resources? Answers to these questions would be helpful when it comes to making decisions about how to allocate library resources for improving reference services Real-time online references services are increasing in number across the country and around the world The hours of operation, the staf ng patterns and the settings in which they are being conducted vary a great deal For instance, some libraries may provide real-time reference only during their hours of operation and staff it with their librarians Other real-time online reference services are provided by commercial enterprises for a fee and are generally not staffed by professional librarians Future studies could be conducted to gather information about both the patrons 571 and the service providers in these various settings Additional studies could compare and contrast the various types of real-time online reference services References Abels, EG (1996), The reference interview, RQ, Vol 35 No 3, pp Allen, B (1995), Evaluation of reference services, in Bopp, RE and Smith, LC (Eds), Reference and Information Services: An Introduction, Libraries Unlimited, Englewood, CO, pp Bushallow-Wilbur, L, DeVinney, G and Whitcomb, F (1996), Electronic mail reference service: a study, RQ, Vol 35 No 3, pp Devlin, B and Burke, M (1997), Internet: the ultimate reference tool?, Internet Research, Vol 7 No 2, pp Eichler, L and Halperin, M (2000), LivePerson: keeping reference alive and clicking, EContent, Vol 23 No 3, pp 63-6 Fishman, DL (1998), Managing the virtual reference desk: how to plan an effective reference system, Medical Reference Services Quarterly, Vol 17 No 1, pp 1-10 Garnsey, BA and Powell, RR (2000), Electronic mail reference services in the public library, Reference & User Services Quarterly, Vol 39 No 3, pp Janes, J (2002), Digital reference: reference librarians experiences and attitudes, Journal of the American Society for Information Science and Technology, Vol 53 No 7, pp Janes, J, Carter, D and Memmott, P (1999), Digital reference services in academic libraries, Reference & User Services Quarterly, Vol 39 No 2, pp Kasowitz, A, Bennett, B and Lankes, RD (2000), Quality standards for digital reference consortia, Reference & User Services Quarterly, Vol 39 No 4, pp Koutnik, C (1997), The World Wide Web is here: is the end of printed reference sources near?, RQ, Vol 36 No 3, pp Lipow, AG (1999), In your face reference service, Library Journal, Vol 124 No 13, pp 50-2 NOLA (2001), AskUsQuestionscom: building cooperative Web-based reference services, NOLA Online, available at: wwwnolanetorg (accessed 20 January 2003) Oder, N (2001), The shape of e-reference, Library Journal, Vol 126 No 2, pp Ross, CS and Nilsen, K (2000), Has the Internet changed anything in reference?, Reference & User Services Quarterly, Vol 40 No 2, pp Sloan, B (2001), Ready for reference: academic libraries offer live web-based reference: evaluating system use, available at: wwwlisuiucedu/~b-sloan/r4r nalhtm (accessed 20 January 2003) Tenopir, C and Ennis, L (2002), A decade of digital reference , Reference & User Services Quarterly, Vol 41 No 3, pp Zumalt, JR and Pasicznyuk, RW (1998), The Internet and reference services, Reference & User Services Quarterly, Vol 38 No 2, pp

8 Appendix Real-time reference service librarian/operator questionnaire 572

9 573

10 574

11 Biographies Louise Kloss has a Master s Degree in Library and Information Science from Kent State University She has been an adult services reference librarian at Morley Library in Painesville, Ohio for the past nine years In addition to providing reference services, she is Morley Library s Database Coordinator; works part-time providing online reference assistance as one of NOLA s AskUsQuestionscom operators and is a member of the Ohio Library Council Louise can be contacted at: Louise Kloss, Morley Library, 184 Phelps Street, Painesville, OH 44077, USA Tel: (440) ; klosslo@oplinlibohus Yin Zhang is an assistant professor at Kent State University School of Library and Information Science She holds a PhD in Library and Information Science from the University of Illinois at Urbana-Champaign Her research interests include information seeking behaviour in the Internet environment, evaluation of information uses and services, and distance learning Her papers have appeared in Journal of the American Society for Information Science and Technology, Journal of Information Science, Online Information Review, and Library Trends Yin can be contacted at: Yin Zhang, School of Library and Information Science, Kent State University, Kent, OH 44242, USA Tel: (330) ; Fax: (330) ; yzhang4@kentedu 575