End-User Searching in Medicine*t

End-User Searching in Medicine*t ABSTRACT Investigation of end-user searching at the New York Hospital-Cornell University Medical Center (NYH- CUMC) revealed that 8% of the physicians surveyed were end users, 63% were interested in learning to search, and 29% were not interested. When training sessions were offered at the Burke Rehabilitation Center, an affiliated institution, 50% of the medical staff attended at least one class, but only 7% of the total staff became frequent searchers. Analysis of the precision and recall ratios of searches conducted by five end users at HYH-CUMC indicated that the best results were obtained by end users who had been taught to search by experienced librariansearchers. The quality of end user searches did not appear to be affected by the "friendliness" of the systems used, the frequency of searching habits, or the length of time that an end user had been searching. INTEREST IN end-user searching, and particularly in end-user searching by physicians, has grown dramatically during the past five years. Prior to 1979, the literature on user-friendly searching was limited to technical discussions on search language [1]. In 1979, however, Meadow made the prediction that the end-user "will eventually take over the terminal" [2]. By 1981, Paper- Chase, a user-friendly, in-house system for searching a subset of Index Medicus, was announced in the New England Journal ofmedicine [3], and the impact of end-user searching of online databases [4] and of online catalogs [5] was being discussed seriously. Commercial database producers were quick to follow with products directed to the end-user market: Sci-Mate [6], BRS/After Dark [7], and Knowledge Index [8]. These systems are now being promoted in articles directed to physicians [9]. *An abbreviated version of this paper was presented on May 27, 1985, at the Eighty-fifth Annual Meeting of the Medical Library Association, New York, New York. tthe research for this paper was conducted in the fall of 1984 at the New York Hospital-Cornell Medical Center, where the author was head of the Reference Department. BY ELLEN H. POISSON, Librarian CIBA-Geigy Corporation Pharmaceuticals Division 556 Morris Avenue Summit, New Jersey 07901 Physicians have also been targeted by the AMA/ GTE system and by the BRS/Colleague medical databases. Another in-house end-user system, minimedline, has been described by Broering [10]. In 1984, the first classes for end users and trainers of end users were sponsored by the National Library of Medicine for the MEDLARS system. In 1985, Mead Data launched MEDIS, a full-text database of key journals in clinical medicine, which is easily searched by using predefined function keys and keyword searching. Thus, the choices and availability of end-user systems have broadened considerably. The current use of these systems provides a way to measure the success of promotional efforts and the practicality of end-user searching in available systems. STUDIES ON END-USER SEARCHING The level of end-user searching actually taking place has been the subject of several recent studies that have focused on the results of training sessions for end users in industry. At the Eastman Kodak Company, Haines reported that of fifteen chemists who attended training sessions, seven (47%) were still searching at the end of one year [11]. At a Kodak subsidiary, however, the success rate was only one in six (17%) for researchers drawn from several divisions. Walton reported in 1983 that the end-user searching rate at the end of a training course at Exxon was eight in eighteen, and that only three (17%) were still searching at the end of one year [12]. Another researcher, Richardson, does not give specific figures for his study of an end-user searching program at the Raytheon Company, but he does state that "overall system usage was surprisingly low" [13]. Leipzig reported relative success in end-user searching for an in-house database of proprietary information at a pharmaceutical company, but low use of MEDLARS databases after training: one in six (17%) [14]. At Derwent's Chemical Reactions Documentation Service, four 293

ELLEN H. POISSON of eight chemists searched at least once following training, and all end-users later averaged one search per month. PURPOSE OF THE STUDY During 1983 and 1984 it became apparent that physicians' interest in end-user searching was increasing. At the Reference Desk at NHY- CUMC, queries on end-user searching were received weekly and sometimes even daily. Access to computers in offices and clinics had increased, and promotion by database vendors was beginning to have an impact. Ideally, end-user searching would bring clinical information into the hands of the physician, making immediate access to the medical literature possible from an office or an emergency room. In this way, pertinent literature would be available when a clinical decision was needed, and the search would be performed by people directly involved in the patient's care who had the expertise to evaluate and apply the results. This study was designed to address the following questions: 1. What percentage of physicians today are interested in end-user searching? Are physi- cians who are interested in end-user searching more likely to be frequent users of online search services? Conversely, are physicians who are not interested in end-user searching more likely to be nonusers of search services? Do end users also use search services, or are they more likely to be nonusers? 2. What percentage of physicians who attend end-user training sessions become end users? How does this compare to studies of end users in fields other than medicine? 3. What are the precision and recall ratios for end-user searches? What factors in the training of end users affect the quality of their searching? This study attempted to compare the professed interest in end-user searching to the actual searching by physicians. The effect of end-user searching on the use of search intermediaries was also examined. Finally, factors that have a positive impact on the quality of end-user searching were identified. PHYSICIANS' INTEREST IN END-USER SEARCHING In order to estimate physicians' interest in computer searching, a random sample survey was con- 8% INTEREST IN END)-U4 ISER SEARCHING _ ---- ------ --- ----- _- - - ---- _ ----I -( T S - - - 29% NOT INTERESTED (19) ---------- --------------------- -- - - -_-_- \~~~~~~3 ---------------_-- w-, ======== ---= ===-=====- INTERESTED. ----------------- =_A -- = = = = _ = = "l1 n = 65 \- 294 FIG. I. Interest in end-user searching.

END-USER SEARCHING IN MEDICINE 30 USE OF COMPUTER SEARCH SERVICES BY RESPONDENTS INTERESTED IN END-USER SEARCHING 25 F- NUMBER OF END USERS 20k 151-10k 5k n=41 FREQUENT OCCASIONAL NEVER USE USE (7) (7) (27) USE OF SEARCH SERVICES FIG. 2.-Use of computer search services by respondents interested in end-user searching. ducted by questionnaire at NYH-CUMC. The sample was a representative group of urban physicians, teachers, practitioners, and researchers from all major specialties. The survey was designed to estimate the percentage of physicians now doing their own searching, the percentage interested in learning to search, and the percentage not interested. The respondents were also asked how often they use search services at the library or elsewhere. Of 139 questionnaires, sixty-five (47%) were returned with usable data. Of the respondents, five (8%) were doing their own searching, forty-one (63%) were interested in learning to search, and nineteen (29%) were not interested (Figure 1). Therefore, 71% of the respondents were either searching or interested in learning to search. Comments added in the margins ranged from "Good idea!!" to "I'm too busy to learn." The questionnaire also asked how often physi- cians used search services at the library or elsewhere. The categories offered were: frequently (once or more in three months), occasionally (less than once in three months), and never. Of the five respondents who were doing their own searching, none reported frequent use of search services, one reported occasional use, and four reported that they never use a search service. The respondents who reported that they were interested in learning to search were more frequent users of search services (Figure 2). Of these forty-one, seven were frequent users, twenty-seven occasional users, and seven never used a search service. The respondents who were not interested in learning to search were also more likely to be nonusers of search services: of nineteen, two were frequent users, twelve occasional users, and five nonusers. In summary, 71% of the physicians responding to this survey were interested in end-user searching to some degree, but only 8% had taken the plunge 295

ELLEN H. POISSON and were actually doing their own searching. The data also show low use or nonuse of search services by end users. END-USER SEARCHING AFTER TRAINING In 1983/1984, training sessions for medical staff were conducted at the Burke Rehabilitation Center in Westchester, N.Y. Questionnaires were sent to attendees to determine how many were searching on a regular basis as of the fall of 1984. The staff at Burke had a particular incentive to learn to search, because there is no librarian on-site. All searches must be requested by mail or phone from the Reference Department at NYH-CUMC. The survey showed that of the sixty staff members, thirty attended at least one training session (Figure 3). Of those who did attend, sixteen (53.3%) had not done any searching, six (20%) had searched less than once in six months, four (13.3%) had searched once in three months or once in six months, and a core group of four (13.3%) were relatively frequent searchers-more than once in three months. The respondents were asked to give comments on searching, and some of these were: "System is easy to use as it is at present." "Has saved an immense amount of time and helped formulate our thinking about a problem." "Since I use it so infrequently, I would have to study process each time." "BRS is a little unfriendly to use occasionally." "Very good source of references for background to research on new areas, but often hard to get the desired literature without also getting a fair number of articles which don't pertain." "I feel that time to develop skills to use computer effectively would not be repaid at this point in time." "System here is tedious and time-consuming, therefore I request search by librarian." In summary, 50% of the professional staff at Burke attended at least one training session, but only 7% of the total staff were frequent end users, a figure similar to the estimated 8% at NYH- CUMC. LEVEL OF END-USER SEARCHING AFTER ATTENDANCE AT TRAINING SESSIONS RARELY SEARCHING 20% 1\i,~ (6) ATTENDED AT LEAST ONE SESSION, BUT HAVE NOT SEARCHED 53.3% (16) OCCASIONAL SEARCHING 13.3% (4) n = 30 296 FREQUENT SEARCHING 13.3% (4) FIG. 3.-Level of end-user searching after training sessions.

END-USER SEARCHING IN MEDICINE 100 F 901- PRECISION = NO. OF PERTINENT ARTICLES RETRIEVED NO. OF ARTICLES RETRIEVED PRECISION 80 I 70 [ 60F 50 F 40-301- 20 F 101 1 A, l1 lzz, 2 END-USERS FIG. 4.-Precision ratios of end users. 3 A 11 --.,, 4 5 FACTORS AFFECTING QUALITY OF END-USER SEARCHING Several end users at NYH-CUMC were given a topic to be searched in MEDLINE and five responses were received. These responses were translated into a single search language and retrievals were measured for precision and recall. (The precision ratio is calculated by dividing the number of pertinent articles retrieved by the total retrieved. The recall ratio is calculated by dividing the number of pertinent articles retrieved by the number of pertinent articles in the database.) The number of relevant references in the database was determined by an experienced searcher. The end users were also asked which systems they routinely search, how long they had been searching, how frequently they search, and how they learned to search. The results were as follows: the precision ratios were quite good, especially for end users 1, 4, and 5 (Figure 4). Retrievals by end users 2 and 3 included irrelevant articles (approximately 40%). The recall ratios showed more dramatic differences in the quality of these searches (Figure 5). End users 1 and 4 still did quite well; they were able to retrieve about 80% of the relevant articles in the database. End users 2, 3, and 5 retrieved only 12-22%. These end users relied on free-text searching; they omitted subject headings, subheadings, and explosions from their strategies. Although this is a small sample, factors that may have affected performance can still be analyzed. The choice of a user-friendly system does not seem to have helped, because BRS/Colleague was used by end user 2, and NLM was used by all of the others. Frequent practice in searching does not seem critical either, because the two end users with the worst results searched the most and the least frequently. The length of time that they had been searching was also not important, because end users 1 and 2 had been searching from one to six months, and the others had all searched for more than six months. The one factor that does seem relevant is how they learned to search. The two end users with the best results were taught to search by librarians who emphasized subject headings, explosions, check 297

ELLEN H. POISSON RECALL = NO. OF PERTINENT ARTICLES RETRIEVED NO. OF PERTINENT ARTICLES IN DATABASE RECALL 1 2 3 4 5 END USERS FIG. 5.-Recall ratios of end users. tags, and subheadings in searching MEDLINE. The others reported that they learned to search by reading the documentation, through the MED- LEARN self-instruction program, and from a database vendor's representative. It appeared that although these end users had mastered system commands, they had not learned to use the controlled vocabulary effectively. These results corroborate Olson's study, which revealed that end-users trained by librarians were able to use the interactive capabilities of MEDLINE more effectively than "untrained searchers" who had not been instructed by librarians [15]. SUMMARY In this study, 71% of the physicians surveyed were interested in end-user searching, but only 7% or 8% were doing their own searching regularly. This rate is lower than the 17-47% range that was noted in the studies of end-user searching in industry reported above. The data also indicate that end-user searching results in low use of search services. In this study, 80% of end users were 298 nonusers of search services, as opposed to 22% of all others. The end users with the highest precision and recall ratios had been taught to use the controlled vocabulary of MEDLINE, as well as the system commands, by an experienced librarian-searcher. IMPLICATIONS These results suggest that librarians can encourage high-quality end-user searching by becoming teachers and consultants for those who want to learn to search. End users should be cautioned against viewing online searching as a panacea, however; effective use of the traditional, printed sources should also be taught. As this study indicates, the quality of end-user searching can vary widely, and one of the greatest dangers is that end users may be unaware of the incompleteness of their searches. They may also become frustrated because their searches yield too many false drops. Because only a small percentage of physicians are end users now, librarians will continue to be search intermediaries for most physicians-for the

END-USER SEARCHING IN MEDICINE time being. Increased user-friendliness of systems, price cuts, full-text databases, and physicians' growing familiarity with and access to personal computers are certain to have an impact on enduser searching over the next few years. Perhaps in the future computerized database searching will become an essential skill in medical education and an integral part of the curricula [16]. This may lead to improved access to the medical literature, changes in patterns of information seeking by physicians [17], and redefined roles for medical librarians. REFERENCES 1. Goldstein CM, Ford WH. The user-cordial interface. Online Rev 1978 Sep;2:269-75. 2. Meadow CT. Online searching and computer programming: some behavioral similarities (or... why end users will eventually take over the terminal). Online 1979 Jan;3:49-52. 3. Horowitz GL, Bleich HL. PaperChase: a computer program to search the medical literature. N Engl J Med 1981 Oct 15;305:924-30. 4. Faibisoff SG, Hurych J. Is there a future for the end user in online bibliography searching? Spec Libr 1981 Oct;72:347-55. 5. Weiss MC. User reactions to online catalogs: an exploratory study. Coll Res Libr 1981 July; 42:295-302. 6. Stout C, Marcinko T. Sci-Mate: a menu-driven universal online searcher and personal data manager. Online 1983 Sep;7:112-6. 7. Janke RV. BRS/After Dark: the birth of online self-service. Online 1983 Sep;7:12-29. 8. Ojala M. Knowledge Index: a review. Online 1983 Sep;7:31-4. 9. Naisawald GV. Your computer puts the literature at your fingertips. Medcomp 1983 Oct;34-8, 72. 10. Broering NC. The minimedline SYSTEM: a library-based end-user search system. Bull Med Libr Assoc 1985 Apr;73(2):138-45. 11. Haines JS. Experiences in training end-user searchers. Online 1982 Nov;6:14-23. 12. Walton KR, Dedert PL. Experiences at Exxon in training end users to search technical databases online. Online 1983 Sep;7:42-50. 13. Richardson RJ. End-user online searching in a hightechnology engineering environment. Online 1983;7:44-57. 14. Leipzig N, Kozak MG, Schwartz RA. Experiences with end-user searching at a pharmaceutical company. In: Proceedings of the fourth National Online Meeting, New York, April 12-14, 1983:325-32. 15. Olson PE. Mechanization of library procedures in the medium-sized medical library: XV. A study of the interaction of nonlibrarian searchers with the MEDLINE retrieval system. Bull Med Libr Assoc 1975 Jan;63(1):35-41. 16. Matheson N, Lindberg DAB. Subgroup report on medical information science skills. J Med Educ 1984 Nov;59 (pt. 2):155-9. 17. Northup DE, Moore-West M, Skipper B, Teaf SR. Characteristics of clinical information-searching: investigation using critical incident technique. J Med Educ 1983 Nov;58:873-81. Received November 1985; accepted March 1986. 299