Transcription

1 TITLE: Current State of Digital Reference in Primary and Secondary Education AUTHOR(s): R. David Lankes PUBLICATION TYPE: Journal DATE: 2003 FINAL CITATION: Current State of Digital Reference in Primary and Secondary Education. Lankes, R. D. (2003). 9(2) D-Lib Magazine KEYWORDS: Digital reference, virtual reference, information systems, research, digital libraries, education, k12

2 Search Back Issues Author Index Title Index Contents D-Lib Magazine February 2003 Volume 9 Number 2 ISSN Current State of Digital Reference in Primary and Secondary Education R. David Lankes, Ph.D. Director Information Institute of Syracuse Assistant Professor Syracuse University's School of Information Studies Visiting Fellow National Library of Canada <rdlankes@ericir.syr.edu> Abstract This paper examines the domain of digital reference services for and by the primary and secondary education community. Data is provided to demonstrate the current understanding of education question types and education users in digital reference. It is believed this data will be of wide utility for digital library builders geared toward primary and secondary users (K-12) such as the International Children's Digital Library [1] and the National Science Digital Library [2]. Digital Reference and Digital Libraries for Education Digital reference refers to Internet-based expert answering services [Lankes, 1999b]. In such a service, a user typically poses a question to a digital reference service through a web form, or a chat interface. An expert (such as a scientist or librarian) uses this input to construct an answer that is both passed back to the user as well as used in some knowledge base or enhancement to a digital library collection. There is a growing body of research and development in digital reference [Janes, 2000; Mardikian and Kesselman, 1995; Tyckoson, 2002; Lagace and McClennen, 1998; Mon, 2000; Ferguson and Bunge, 1997], and digital reference services are being implemented within the context of digital libraries such as the National Science Foundation(NSF) National Science Digital Library [National Research Council, 1998]. Much is known about use of digital reference in primary and secondary education (in this article, the term "education" will refer simply to primary and secondary education), and this knowledge has utility not only to digital reference services aimed at this population, but to the larger digital library community as well. Specifically, the author argues that services aimed at primary and secondary users represent a revelatory case for digital reference for three reasons; digital reference services for education are: 1. Well researched: Much of the initial research in digital reference was founded in the AskA community, which saw education as a primary audience. 2. Well funded: Education digital reference services such as AskERIC, Dr. Math, KidsConnect, and the Virtual Reference Desk's Learning Center have received relatively substantial (though not always long-lived) funding. 3. Well established: Many education digital reference services have established and consistent funding. Current State of the Art In order to present a picture of digital reference for education, the author first presents two major types of services in the education domain. Each type is then illustrated with an exemplar service. Types of Digital Reference Services in Education There are two obvious, though often overlapping, categories of digital reference services in education: library-based services, and AskA services. AskA services can be further divided into general services that may be of use to the education community as part of a more general mission (such as Ask Joan of Art [3], an AskA service that answers questions concerning American art for anyone who asks, but is particularly useful in art

3 education) and services targeted squarely at the education community (such as AskERIC(sm) [4], though it covers all levels of education including higher and continuing education). The author will concentrate on education AskA services for this paper. Library Reference For the purposes of this discussion, "library reference" refers to digital reference services either centered in a public, academic, school or special library, or with primary reliance on library programs. With the advent of digital reference, a great number of libraries are now offering reference service to remote patrons [Janes, 2000]. These services take a variety of forms, from systems, to real-time chat systems. In the library context, digital reference is referred to as virtual reference, e-reference, networked reference, live reference, online reference and even chat reference. While some in the community make a distinction in the mode of delivery and the synchronous nature of the service offered, most agree that these are all part of a single larger concept of digital reference. The library reference community also provides the most in-depth discussion of policy, evaluation [McClure and Lankes, 2001] and the largest set of documented digital reference services (as opposed to the body of systems and development work out of the AskA community discussed later). Much of this work is encapsulated in the proceedings of the annual Virtual Reference Desk (VRD) Conferences [Virtual Reference Desk, 2002], which have a strong library emphasis. In fact, this article and the Digital Reference Research Symposium were outgrowths of the VRD conferences and other work. As a result of this intense interest in digital reference by the library community, several large-scale digital reference projects are available for use by the research and scholarly community. The Collaborative Digital Reference Service (CDRS) spearheaded by the Library of Congress and that has evolved into the QuestionPoint service run by OCLC in cooperation with the Library of Congress, certainly demonstrates the breadth of library-based digital reference services spanning public, academic and international libraries. The National Library of Canada's recent introduction of Virtual Reference Canada [5] to work with Canadian digital reference services also promises to be a major source of digital reference activity and development. Other prominent digital reference efforts in the library world include KnowItNow [6] from the Cleveland Public Library, the 24/7 Reference service [7] that acts as a statewide digital reference network for the State of California, and the recent efforts of the State Library of Washington. Also of interest to researchers in digital reference are digital reference vendors in the library domain including LSSI's Virtual Reference Service [8]. One special case that should not be overlooked is the Internet Public Library [9], for while it is not based in a library setting (it is part of the School of Information at the University of Michigan), it has its roots and traditions firmly planted in the library community. Library Reference Exemplar: KidsConnect While many library services support the education community (of course, academic libraries serve a higher education population and public libraries answer questions of students), few target primary and secondary education exclusively. One exception is the KidsConnect service. KidsConnect is a question-answering, help and referral service to K-12 students on the Internet [KidsConnect, 2002; Bennett, 1998]. It is a project of the American Library Association's American Association of School Librarians (AASL). KidsConnect has three missions: The first is to educate school library media specialists in the use of the Internet and digital reference as part of the larger ICONnect project. The second is to promote information literacy in students through digital reference [Mancall et al., 1999]. The third is to promote local school libraries (and school library media specialists) as valuable sources of information and instruction. The KidsConnect model uses a large number of volunteer school library media specialists (primarily in the United States). Each volunteer is trained using an in-depth mentoring process, then answers questions (ranging from one question a day to one a week). The digital reference transaction is conducted through and web forms. Data from the KidsConnect service provides valuable insight into the types of students using digital reference services as well as the types of questions they ask. The service has been widely advertised to schools, particularly to teachers and school library media specialists. This advertising has been done through the professional association for school library media specialists (AASL), as well as through the Internet. The data presented in Figures 1-5 are from ; however, more recent data presented in Figures 6-9 are used to estimate the current validity of the earlier numbers. Figure 1 shows the number of questions answered by KidsConnect for the years :

4 Figure 1: Number of KidsConnect Questions These numbers are very much in line, though on the high end, with current numbers of library-based digital reference services as reported at recent library meetings, including the 2002 American Library Association conference. Figure 2 shows how these questions were distributed across differing student and adult populations: Figure 2: KidsConnect Users by Education level These figures demonstrate a rough equivalence between primary (elementary and middle school) and secondary education (high school). The low number of users identified as "adult" is explained by both the focus of the KidsConnect service (K-12 students), but also that any questions KidsConnect received from teachers were routed (sent to) the AskERIC service. A more interesting finding, however, was the gender distribution of the questioners as seen in Figure 3: Figure 3: Gender of Kids Connect Users One interesting finding of the KidsConnect staff was the prominence of girls asking questions. While many hypotheses were put forward to explain this situation ( providing a "safer" environment to ask questions than the well documented male dominated classroom, for example), no formal research was conducted to follow up on this finding. The other interesting finding from the KidsConnect data related to the topics or subjects of the questions asked

5 of KidsConnect. The KidsConnect team utilized a "Subject Line Analysis" technique whereby the subject lines of a random sample of questions were examined and classified inductively into a subject scheme. If the subject lines were felt to be uninformative (they did not indicate topicality but instead were words or phrases like "Hello" or "Please help") the underlying question was examined. The results of this analysis are shown in Figure 4: Figure 4: Subject Distribution of Questions It is clear from this figure that questions on the topic of science constituted the bulk of questions received. In order to provide a clearer picture of this category, the analysis was further refined by "type of science questions", as shown in Figure 5: Figure 5: Further Analysis of Science Subjects Data such as this should prove of great use to new digital reference services geared towards education, most notably the NSF's National SMETE Digital Library [NSDL, 2002]. As mentioned before, these statistics represent somewhat dated analysis (4 years old). In 1999, operation of the KidsConnect service moved from Syracuse University to Drexel University (the previous statistics are based on Syracuse data). Syracuse then transferred much of the staff and processes of KidsConnect into the Virtual Reference Desk Learning Center. This project had a slightly different aim; it had a broader focus and also worked in a network of AskA services with general foci. However, the main concentration of the service was still on school library media specialists answering questions from the education community. Statistics from the VRD service show a strong correlation between older KidsConnect statistics and more recent VRD usage. For example, Figure 6 shows the user populations of the VRD service:

6 Figure 6: Grade Level of VRD Patrons Note the higher "adult" population (as compared with Figure 2) reflecting the broader focus of the VRD Network members. However, with this result removed, the distribution in primary and secondary education remains roughly equivalent to the earlier data, with a greater number of "middle school" questions. Also note in Figure 7 that science questions still dominate the service: Figure 7: Subject Area of VRD Questions Once again, Figure 8 provides a more fine-grained analysis of science questions: Figure 8: Science Subject Areas of VRD Questions This distributions seems to hold over the three most recent years of the VRD service (as seen in Figure 9):

7 Figure 9: VRD Subjects over Time From these more recent statistics, it seems difficult to argue that there has been a massive shift in the types of education users asking questions or in the types of questions they ask. What is also clear from analyzing these two services is that the library community has many contributions to make to the digital reference research agenda specifically with respect to education as well as to digital reference research in general. It is also clear the library community contains large-scale digital reference efforts that make excellent research environments capable of being utilized in the search for generalizable knowledge. Education AskA Services The second progenitor of current digital reference systems is AskA services. AskA services take their name from expert question and answer services tending to adopt names such as "Ask A Scientist" [10] and "Ask A Volcanologist" [11] [Lankes, 1999b]. These services tended to originate without interaction with formal library systems and emphasized topical expertise (as opposed to process expertise such as a librarian's ability to search for information). A fuller picture of AskA services can be drawn from two studies conducted by Lankes and White [Lankes, 1999b; Lankes, 1999c; and White, 1999]. Lankes presents an in-depth analysis of the structure and commonalities of "exemplary K-12 digital reference services." Specifically this study sought to: Build and apply a conceptual framework based on complexity research, literature and the researcher's experience; Use this conceptual framework to empirically describe how organizations, specifically K-12 digital reference services, build and maintain services in the dynamic Internet environment; and Seek commonalties across these descriptions. The outcome of this study included detailed "blueprints" and a tuned framework of AskA services grounded in complexity theory, as seen in Figure 10: Figure 10: Tuned Framework of K-12 Digital Reference Services White developed an analytical framework based on systems for evaluating AskA services. This framework was then applied to a variety of 11 services (including library-based services). Unlike library digital reference services that to this point have seen modest usage, AskA services, in general, have begun with large usage and have experienced continuing dramatic increases. The most recent Virtual Reference Desk survey of AskA services done in 1999 demonstrates this. Survey results in Table 1 show an average 44% increase in use of these asynchronous services from 1997 to 1998, with an average answer rate of 77% in 1998 [Lankes and Shostack, forthcoming]. Table 1: Virtual Reference Desk Survey of AskA Service Usage

8 GEM Subject Service Name Questions Received Per Week in 1997 Multiple Subject The Arts General Education General Reference Health Language Arts Mathematics Religion Science Astronomy Engineering General Science Geology Natural History Questions Received Per Week in 1998 Percent Difference in Questions Received ScienceNet % 100% National Museum of American Art Reference Desk % 75% AskERIC % 100% KidsConnect % 100% The Internet Public Library % 62% Ask the Dentist % 1% Ask the Diabetes Team % 100% Dr. Greene's HouseCalls % 10% The ESL Help Center % 100% Ask Dr. Math % 35% Ask the Math Tutor % 75% Ask an Amish Expert % 100% Ask an Astronomer % 50% Ask a NASA Scientist % 60% Ask the Space Scientist Ask Professor Construction The MAD Scientist Network % 70% % 90% % 88% Ask-An-Earth-Scientist % 65% Ask a Geologist (Geological Survey of Canada) % 100% Ask a Hydrologist % 100% Ask a Volcanologist % 100% Percent of Questions Answered in 1998

9 Oceanography Social Studies Dino Russ's Lair % 95% Ask Jake, The Sea Dog % 100% Ask Shamu % 100% Ask the Harkster (Canada) % 50% Total Questions % Averages % 77% Compare these statistics to those of the libraries studied as part of McClure and Lankes Quality Study [Mcclure and Lankes, 2001], "In all cases the volume of digital reference questions is low, ranging from 3 to 33 per day" [Gross et al., 2002]. This study covered a range of libraries in terms of size and scope (academic, public, federal, state). One result of the large volume encountered by AskA services has been an emphasis on process, software development and automation. Whereas many library services have quickly adopted real-time technologies in which one-to-one interactions require full human intervention, AskA services have looked to asynchronous technologies. (At least this has been so at their onset. See Figure 11 for the distribution of questions received by AskERIC by mode of digital reference as an example of the predominance of asynchronous means. Note that "web" and " " are both asynchronous modes.) AskA services have also looked for means of shunting users to resources. (See Lankes for a more detailed discussion of AskA services and their architectures [Lankes, 1999b].) These run the gambit from sophisticated techniques such as automated searching of previously asked questions (as in the MadSci [12] service), to forcing users through a list of frequently asked questions before they are able to submit a question (as in the Ask A Volcanologist service). Figure 11: Mode of Asking AskERIC a Question AskA services have also tended to develop more in terms of software and systems. Early examples include Ask Dr. Math [13], the MadSci Network, and How Things Work [14]. Though there are excellent examples of software development in the library arena [Meola and Starmont, 2002], library services have by and large adopted software from the help desk and e-commerce community, such as LSSI and 24/7 Reference's use of egain and the common use of LivePerson and NetAgent. While this may be changing, AskA services still remain a hot bed of systems development. Another common attribute with AskA services is their attention to the primary and secondary education community. In the case of some services, this attention is part of a larger view of the general Internet population, but in many cases, it is a special attention where education is foremost and the general population is welcome as well. This can be seen in Dr. Math and MadSci Networks. It can also be seen in services, such as AskERIC, which focus on education professionals. Education AskA Service Exemplar: AskERIC While the KidsConnect discussion sheds light on digital reference use by primary and secondary education students, AskERIC can shed light on use of digital reference by education professionals. AskERIC is a project of the U.S. Department of Education's ERIC program. It was initiated and is still operated by the ERIC Clearinghouse on Information & Technology (though nearly all ERIC components (Clearinghouses,

10 ACCESS ERIC, the ERIC Processing Facility and even the parent institution of ERIC, the National Library of Education) are involved in answering questions). AskERIC has two primary components: a question/answering service staffed by ERIC library and education professionals (see Figure 12 for the volume of questions), and a virtual library of lesson plans, which contains pointers to reviewed sites on the Internet and an archive of previously asked questions. A more in-depth description, though slightly dated, can be found in the author's dissertation [Lankes 1999b]. Figure 12: Volume of AskERIC Questions over Time The purpose of AskERIC is to answer questions related to all areas of the education process. The emphasis on education professionals can be seen in AskERIC's mission as well, see Figure 13, by AskERIC's users. Figure 13: capacity in Which AskERIC Users Asked Questions This distribution of users, with the majority being K-12 teachers followed by graduate students (pre-service educators are traditionally heavy users of any ERIC service), is in line with AskERIC's stated mission: AskERIC is a personalized Internet-based service providing education information to teachers, librarians, counselors, administrators, parents, and anyone interested in education throughout the United States and the world. [AskERIC, 2002] In fact, AskERIC explicitly does not answer "homework help" questions: Thank you for visiting the AskERIC Web site! If you are a K-12 student with a homework question, AskERIC may not have the resources to respond to your question. AskERIC is designed to provide education information to teachers, librarians, counselors,

11 administrators, parents, students, and others throughout the United States and the world. Our focus is not on the specific things you are learning in school; instead, we specialize in research and ideas about how students of all ages learn best. As an example, we can respond to a question such as "What is the best time of day to teach math?", but not "What is the formula to determine the radius of a circle?". If you are looking for information in other specific subject areas or need homework help, you probably won't find AskERIC very helpful. Instead, you may want to investigate the following sites which are designed specifically for students. [AskERIC, 2002a] All student questions received by AskERIC are forwarded to other services such as the Virtual Reference Desk. What can one determine about AskERIC users besides their educational roles? First, one can determine the education level about which users were asking (a K-12 teacher was asking a question about high school, for example) as seen in Figure 14: Figure 14: Level Focus of AskERIC Questions One can also analyze the nature of the questions being asked by the professional community. AskERIC user surveys provide the anticipated use of the information gained as seen in Figure 15: Figure 15: Planned Use of AskERIC Responses Using subject line analysis once again, Figure 16 shows question types identified in AskERIC questions:

12 Figure 16: Subjects of AskERIC Questions Figure 17 shows the relative stability of this question distribution over time: Figure 17: Subjects of AskERIC Questions over Time In Figure 16 and Figure 17, "subjects" refers to particular topics or academic disciplines taught in the classroom. (Note: information from AskERIC responses may be used in higher and continuing education contexts as seen in Figure 18 where 18% of answers were intended for higher or adult education.)

13 Figure 18: Breakdown of "Subjects" in AskERIC Questions Figure 19 shows the relative stability of these subjects over time: Figure 19: AskERIC Question Subjects over Time Of particular interest in Figure 19 is the predominance of "language arts" as a topic for educators versus "science" for students, as seen in Figure 4 of the KidsConnect sample. One possible reason for this difference may be the abundance of science material on the Internet (particularly education-related science material) versus instructional resources in language and English instruction. Aside from the information AskERIC provides on digital reference use by education professionals, it also provides an exemplar of reference authoring [Lankes, 2001]. Reference authoring refers to the capture of information in the reference process and the transformation of this information into resources that can be used outside the reference process as part of a larger digital library context. This authoring process can be from the simple, say the creation of frequently asked questions on a web site, to the complex, say the creation on the MadSci knowledge base. The heart of the AskERIC website consists of a resource collection: In response to questions we've received at AskERIC, our network information specialists have compiled over 3000 resources on a variety of educational issues. This collection includes Internet sites, educational organizations, and electronic discussion groups. [AskERIC, 2002b] This resource collection acts not only as a set of Internet links for end-users, but for AskERIC digital reference specialists as well. As digital reference specialists constantly comb over this collection of Internet resources, ERIC citations, discussion groups and more, they are also finding new resources to add and old resources to delete. This means that it is the digital reference process itself that is used as collection development, annotation and expert review. AskERIC is only one example of AskA services geared specifically to the education community. It does, however, serve as a revelatory case. In the AskERIC exemplar we see the predominance of asynchronous technologies, high-volume usage, and the interconnection of the reference process with systems and digital libraries. Conclusion Digital reference for primary and secondary education has a rich and well-documented tradition. It serves as a revelatory case for other digital reference research and provides valuable insight into digital libraries that serve the education community as well as other communities. What is apparent from this small examination of digital reference in the education context is that all levels of education use digital reference services and education questions, while covering a broad range of topics, concentrate most heavily on science (in the case of students) and language arts (in the case of education professionals). Also apparent is the usefulness of education digital reference services as research environments. AskA services and library reference services alike hold large data sets of question and answer transactions. These data sets can be used to evaluate how questions are asked, what topics are of interest to the education community, and what language is used by the education community, as well as used to examine myriad other facts. Some of these data sets are publicly available on the Internet, while others are proprietary due to privacy concerns. From this examination of digital reference services, some methodological techniques can be added to the digital

14 reference research discussion. First among these is the concept of subject line analysis. This technique seems to provide excellent exploratory power and may provide a rapid way to compare question types across services. Lastly, analysis of digital reference services targeted towards the primary and secondary education community (or at least the study of these services) provides a wealth of models, theories and frameworks that can be brought to bear in future research. From The Lankes/Sutton Framework, the General Digital Reference Model (resulting from Lankes' complexity framework) to White's evaluative framework, there are rich analytic tools that can be used in the broader digital reference and digital library domain. Notes [1] International Children's Digital Library, < [2] National Science Digital Library, < [3] Ask Joan of Art, < [4] AskERIC, < [5] Virtual Reference Canada, < [6] KnowItNow, < [7] 24/7 Reference, < [8] LSSI's Virtual Reference Toolkit, < [9] Internet Public Library, << [10] Ask A Scientist, < [11] Ask A Volcanologist, < [12] MadSci, < [13] Ask Dr. Math, < [14] How Things Work, < References [ALA] American Library Association Presidential Committee on Information Literacy. (1989). Final report. Chicago: Author. (ED ). [AskERIC, 2002] AskERIC (2002). About AskERIC. (Online) < [AskERIC, 2002a] AskERIC (2002). AskERIC Student Information Page. (Online) < [AskERIC, 2002b] AskERIC (2002). Welcome to AskERIC. (Online) < [Bennett] Bennett, B. (1998). Pilot testing the KidsConnect service. In Lankes, R. and Kasowitz, A. (ED), AskERIC starter kit: How to build and maintain digital reference services, (pp ). Syracuse, NY: ERIC Clearinghouse on Information & Technology, Syracuse University. [Bristow] Bristow, A. "Academic Reference Service over Electronic Mail." College & Research Libraries News, 53 (1992): [Collier] Collier, M. (1997). International Symposium on Research, Development, and Practice in Digital Libraries. [Ferguson and Bunge] Ferguson, C. D. and C. A. Bunge. "The Shape of Services to Come: Values-based Reference Service for the Largely Digital Library." College & Research Libraries, 58 (1997): [Gross, et al.] Gross, M., McClure, C., Hodges, R., Graham, A., and Lankes, R. (2002). Phase II: Site Visit Summary Report. [Janes] Janes, J. (2000). Current Research in Digital Reference. VRD Proceedings. (Online) < [KidsConnect] KidsConnect (2002). ICONnect - KidsConnect. (Online)

15 < [Kindler] Kindler, A. (2002). Table. In Padolsky, D. "How many school-aged limited English proficient students are there in the U.S.? (Online) < [Kresh] Kresh, D. (2000). Offering High Quality Reference Service on the Web. D-Lib Magazine, 6(6). (Online) < [Lankes and Sutton] Lankes, R. David and Sutton, Stuart A. (1999). "Developing a Mission for the National Education Network: The Challenge of Seamless Access." Government Information Quarterly. 16(2). [Lankes, 1999a] Lankes, R.D. (1999a). "The Virtual Reference Desk: Question Interchange Profile." White Paper for the Virtual Reference Desk. ERIC Clearinghouse on Information & Technology; Syracuse, NY. [Lankes, 1999b] Lankes, R. (1999b). Dissertation: Building & Maintaining Internet Information Services, Syracuse University. [Lankes, 1999c] Lankes, R. D. (1999). "AskA's Lesson Learned from K-12 Digital Reference Services." Reference & User Services Quarterly. 38(1). [Lankes and Shostack] Lankes, R. and Shostack, P. (Forthcoming) "The Necessity of Real-Time: Fact and Fiction in Digital Reference Systems." Reference and User Services Quarterly. [Lankes et al.] Lankes, R. David, Collins, J. and Kasowitz, A. S. (Eds.). (2000). Digital Reference: Models for the New Millennium. New York: Neal-Schuman. [Lankes, 2001] Lankes, R. D. (2001). "Creating a New Reference Librarianship." VRD Proceedings. (Online) < [Legace and McClennen] Legace, N. and M. McClennen. (1998). "Questions and Quirks." Computers in Libraries, 18(2) [Online.] (Online) < [Lewis] Lewis, M. (2001). "Faking it." The New York Times Magazine, New York; July 15, [LITA] LITA (1999). Top Tech Trends. (Online) < [MadSci] MadSci Circumnavigator, < [Mancall et al.] Mancall, J. and Stafford, B. and Zanger, C. (1999). ICONnect: A Snapshot of the First Three Years. Knowledge Quest. 28(1), p [Marchionini] Marchionini, G. (1999). Augmenting Library Services: Toward the Sharium. Proceedings of the International Symposium on Digital Libraries 1999 (Tsukuba, Japan, September 28-29, 1999) < [Mardikian and Kesselman] Mardikian, J., and Kesselman, M. (1995). Beyond the desk: Enhanced reference staffing for the electronic library. Reference Services Review, 23 (1), p [McClure and Lankes] McClure, C. and Lankes, R. D. (2001). Assessing Quality in Digital Reference Services, OCLC. Research Project (Online) < [Meola and Starmont] Meola, M. and Starmont, S. (2002). Starting and Operating Live Virtual Reference Services: A How-To-Do-It Manual for Librarians. New York: Neal-Schuman. [Mon] Mon, L. (2000). "Digital Reference Service." Government Information Quarterly, 17(3) [NSDL] NSDL (2002). National Science, Technology, Engineering, and Mathematics Education Digital Library (NSDL) (Online) < < [National Research Council] National Research Council (1998). Developing a Digital National Library for Undergraduate Science, Mathematics, Engineering, and Technology Education. NRC workshop, August 7-8, (Online). Washington, D.C.: National Academy Press. < (ED425928). [Still and Campbell] Still, J., and Campbell, F. (1993). Librarian in a box: The Use of electronic mail for reference. Reference Services Review, 21(1), pp [Tyckoson] Tyckoson, D. (2002). On the Desirableness of Personal Relations Between Librarians and Readers: The Past and Future of Reference Service. The Future of Reference Papers, (Online) < [White] White, M. D., Ed. (1999). Analyzing Electronic Question/Answer Services: Framework and Evaluations

16 of Selected Services. ERIC Document ED [Virtual Reference Desk] Virtual Reference Desk (2002). VRD Conferences. (Online) < Copyright R. David Lankes Top Contents Search Author Index Title Index Back Issues Editorial Guest Editorial Next Article Home the Editor D-Lib Magazine Access Terms and Conditions DOI: /february2003-lankes