Crosswalks: the Path to Universal Access?

Transcription

1 Introduction to Metadata Pathways to Digital Information Crosswalks: the Path to Universal Access? Introduction Mary Woodley Access to the universe of online resources has now become the goal of many institutions that manage information resources. In the past few decades, information-rich institutions (governments, libraries, archives, museums, businesses) have transitioned from in-house manual systems to automated systems in order to provide more efficient means to control their collections. Often these automated systems have been locally designed without any reference to community standards of description or information sharing through networks. Until recently, these legacy systems existed in isolation, serving only specialized knowledge communities. Each system maintained its own structural design for fields of access, description and vocabulary control. In some cases, the only user was the information specialist who served as the mediator for any queries. The advent of the Internet and the exponential growth in electronic resources has increased users' demand for the ability to search across many different metadata structures simultaneously. This has motivated institutions either to convert their metadata to a format more readily accessible, or to provide a single interface to search many heterogeneous databases at the same time. 1 Whether the plan is to design search interfaces or to convert data to a new standard, the first step is to analyze the information elements in each database and correlate the discrete information fields in the different databases that have the same or similar meaning. This is sometimes referred to as metadata mapping or semantic mapping. Crosswalks are the visual representations or maps that show these relationships. 2 Mapping supports the ability of a search engine to query fields with the same or similar content in different databases; in other words it supports semantic interoperability. Crosswalks are not only are important for supporting the demand for one-stop shopping, or cross-domain searching, they are instrumental for converting data in one format to another format that is more widely accessible. Conversion Projects Conversion projects transfer metadata fields or elements from one schema into another. Institutions need to convert data when upgrading to a new system because the legacy system has become obsolete or when the institution has decided to provide greater access to their information. This is accomplished by mapping the structural elements in the older system to those in the new system. In practice, often there is not a 1-to-1 relationship Mary Woodley, page 1 of 10

2 between all of the fields in the two systems, which makes the process of converting data from one system to another more complex. Data fields in the legacy database may not have been well defined or may contain a mix of types of information. In the new database, this information may reside in separate fields. Identifying the unique information within a field to map to a separate field may not always be possible and may require manipulating the same data several times before migrating it. Some of the common misalignments between schemas include 3 : 1. A concept in the original database does not have a perfect equivalent in the target database; e.g., CDWA has a field for Creation-Creator-Identity- Nationality/Culture/Race, whereas Dublin Core does not have a field with the same exact meaning. Since Subject is a much broader category, this is a fuzzy match. 2. Data that exists in one field in the original schema may exist in separate fields in the target database. For example, the CDWA Creation-Place concept may appear in the Subject element in Dublin core as well as in the DC element Coverage. 3. Data in separate fields in the original schema may be in a single field in the target schema; for example, in CDWA, the birth and death dates for a creator are recorded in the Creator-Identity-Dates, as well as in separate fields all apart from the creator s name. In MARC, both dates are a subfield in the string for the author s name. 4. Information in one schema may reside in a field that is indexed, whereas it is only free-text descriptive information in the other schema. 5. There is no field in the target schema with an equivalent meaning, and unrelated information may be forced into the same bucket. 6. In only a few cases does the mapping work equally well in both directions. (See number 2 above.) The first crosswalk presented here goes in a single direction: CDWA was analyzed and the other data systems were mapped to its elements. However, there are types of information that are recorded in MARC that are lost in this process; for example, the concepts of publisher and language are important in library records, but are less relevant to CDWA. Below are two examples of conversion projects that illustrate the complexities of creating crosswalks in order to transfer data to a new schema. Museum Information 4 The Consortium for the Computer Interchange of Museum Information (CIMI) was founded in 1990 to promote the creation of standards for sharing cultural information electronically. 5 Like library data, museums legacy data often exists in databases that can be accessed only locally or through a limited network. As a means to provide access to museum information through the World Wide Web, in 1998 CIMI designed a project to map museum data to Dublin Core Simple, or DC Unqualified, 6 a standard that is still in development to enhance retrieval of Internet resources. Dublin Core consists of 15 core elements, some of which may be refined with qualifiers. 7 Dublin Core, as its name implies, was designed to support the most basic information needs and not those of specialized knowledge institutions. The element/qualifier set will be extended as Mary Woodley, page 2 of 10

3 information needs change. The Dublin Core community will approve new elements or qualifiers that the general information community would find useful and implementors may register elements and qualifiers developed to support special interests. The restrictions on the number and types of elements and qualifiers designated as Dublin Core is to ensure interoperability between the various applications. In the initial testbed, CIMI migrated all the public data in museum records to the set of 15 core elements, unqualified. When one compares the number of elements available in CDWA and the CIMI schemas 8 with the Dublin Core set, it becomes obvious that the level of granularity, or refinement, of information is not equivalent. Several fields in the CIMI schema were mapped to a single element type in DC Simple. For example, DC Simple has a single element for Date. However, there are multiple dates associated with a cultural item that are significant: the date the object was created; the date that the item was acquired or collected; the date that the object was cataloged; etc. Mixing two totally different types of dates into a single element may lead to chaotic search results. Another problem faced by the testbed was how to deal with Subject and Coverage. Initially, there was not a clear distinction in DC Simple between the two elements. CIMI s solution was to duplicate Subject terms in the Coverage element. This resulted in phrases such as Baroque dance being added to the Coverage element. However, Coverage only deals with temporal and spatial characteristics; it is therefore inappropriate to include topical terms like Baroque in this DC element. As a result of these testbeds, CIMI published its Best Practices document. 9 The CIMI conversion is an excellent example of how difficult it is to map data that resides in very specific and narrowly defined fields to a system that lacks the same depth or coverage: in some cases, information is added to inappropriate elements. This reduces the interoperability of the museum data in the Dublin Core standard. CONTENT Title Title.Alternative Description Description.Abstract Description.TableOfContents Dublin Core Elements and Qualifiers as of May, 2000 INTELLECTUAL MANIFESTATION PROPERTY Creator Date Date.Created Date.Valid Date.Available Contributor Language Subject Publisher Identifier Relation Relation. IsVersionOf Relation.HasVersion Relation.IsReplacedBy Relation.Replaces Relation.IsRequiredBy Relation.Requires Rights Format Format.Extent Format.Medium Date.Issued Date.Modified Mary Woodley, page 3 of 10

4 Relation.Requires Relation.IsPartOf Relation.HasPart Relation.IsReferenceBy Relation.References Relation.IsFormatof Relation.HasFormat Source Coverage Coverage.Time Coverage.Place Type SCIPIO Auction Catalogs 10 In 1980, the Research Libraries Group (RLG) created a special database for the cataloging of auction catalogs. Because of the unique access and description needs for auction catalogs, their bibliographic records resided in a separate file from the Books file in Research Libraries Information Network (RLIN). 11 Initially, three institutions contributed to the file, 12 twenty institutions now participate. The cataloging standards, authority records and structural format for these records differed significantly from standard library cataloging. Nor were these records transportable to library online catalogs or other database systems. Access was strictly from the RLIN SCIPIO file. The SCIPIO Advisory Group met in 1997 to form a task force to review RLG s plan to migrate bibliographic records for auction catalogs to the MARC format. The charge of the task force was to review the proposed mapping of the structural elements of the auction catalog to the structure of MARC records. Although the structural elements would migrate, SCIPIO s cataloging rules and authority control have remained the unchanged. 13 After a successful conversion of the original SCIPIO file to MARC, in 1997 the Getty Research Institute (GRI) became the first library to download and integrate the auction catalog records into their library online public access catalog (OPAC), known as IRIS. Over 100,000 of these records were downloaded into IRIS. From their Collections Integrated Catalog page ( the end-user is given the option to search the entire database with auction catalogs, without auction catalogs, or to search only the auction catalogs. 14 The result has been a partial success; patrons can now more easily find the auction catalogs in the GRI s holdings and the catalogs can now be part of the automated circulation process. On the downside, the numerous generic titles of auction catalogs have increased the number of irrelevant hits when searching by keyword. A source of tension is the differences in the authorized forms of personal and corporate names used by the different cataloging schemas. For example, for cataloging auction catalogs, it is important to know the specific auction house (e.g., Christie's, New York). But in the Library of Congress Name Authority File, 15 the authorized form for this auction house is Mary Woodley, page 4 of 10

5 Christie, Manson & Woods International Inc., with the specific houses as crossreferences. In SCIPIO, the names of sellers lack authority control and may be in either natural word order or inverted order. Thus, although it is possible to map the structural elements, the differences in the form of the content reduce the semantic interoperability within the same database. The SCIPIO/MARC migration also illustrates other problems that can arise in the crosswalk/migration process. For some fields there was not a good 1- to-1 fit between the original SCIPIO format and the MARC format; that is, there was a lack of full semantic interoperability. The task force had many discussions concerning the most appropriate field in which to migrate certain information, both descriptive elements and access fields: date of sale, seller and auctioneer names not under authority control, Lugt/Lacour number, sale code, and price list inclusion. Most of this information also provides important access points for searchers. Below is a selection 16 of data mapping from the original SCIPIO format, to a MARC format, to the local OPAC, IRIS. Original SCIPIO fields SCIPIO MARC Migration in a single direction LOCAL OPAC - IRIS NOTES ID Unique record numeric ID AD 008/ /00-05 Originally date of input inform mm-dd-yy converted to yymmdd in MARC DS $c Date of Sale in SCIPIO migrated to Date of Event in MARC; Format in SCIPIO: mmddyy; Format in MARC: yyyymmdd; converted documents lacked date in 260; some libraries added retrospectively and to current cataloging Lancour/Lugt numbers is a call number field in MARC; 024 Other standard number identifier; 500 with Sale code: added Corporate Main entry Title $a $a 518 Place of sale added to note; older records do not have 260$a Place of publication Subject; 650 in MARC is under authority control; 653 is free form Added corporate entry, uncontrolled form Mary Woodley, page 5 of 10

6 Crosswalk for Selected Fields Illustrating the Mapping of SCIPIO Fields to MARC and From RLIN to Local OPAC Searching Federated Searching The number of metadata standards is growing and it is unrealistic to think that every system can be converted to a common standard that will satisfy domain-specific needs. An alternative is to maintain the separate databases which have been developed to support the needs of specific communities, and offer a search interface that allows the user to search across the various heterogeneous databases simultaneously. The results may be displayed either by categories, or integrated. Having the choice of searching a single database or multiple databases allows users to take advantage of the specialized indexing and controlled vocabulary of a single database or to cast a broader net. There are several advantages of a single gateway or portal to information. Users are not always aware which of the many databases to which they have access will provide them with the best information. Libraries have attempted to list databases by categories and provide brief descriptions; but users tend not to read lists, and this type of segregation of resources neglects the interdisciplinary nature of research. Few users have the tenacity to read lengthy A-Z lists of databases, or to ferret out databases relevant to their queries when they are buried in lengthy menus. 17 Many of the issues in creating crosswalks to support searching federated repositories also arise in the process of data conversion: there are no corresponding fields for certain information, fields are not uniformly indexed; fields only share a fuzzy semantic match; the controlled vocabularies for subjects or descriptors do not overlap and are not mapped, and so on. There are also system problems that may arise in conversion projects. For example, not every server will interpret the query from the individual clients in the same manner; different database systems handle Boolean operators, 18 truncation, 19 and keyword versus phrase searching differently; and so on. Research and development in cross-domain searching has been spearheaded by the digital library initiatives and commercial vendors that provide access through subscription to aggregates of specialized databases. Two approaches have been prevalent in the move to provide one-stop shopping : federated searching and using the Z39.50 protocol. Two current digital library projects in California provide access to union catalogs of library holdings that is, the holdings of all the campus libraries in their university system, full-text electronic books and journals, as well as public accessible and subscription electronic indexing databases and Internet resources. Both SearchLight (University of California) and Pharos (California State University) provide a single portal to information through the Z39.50 protocol. In these portal environments, a Z39.50 interface queries multiple Z39.50-compliant databases using standardized indexes. However, the mapping of specific fields to specific indexes does not take into account whatever special indexes may exist locally; nor does it consider any of the local indexing decisions. The University of California and California State University projects provide Mary Woodley, page 6 of 10

7 only keyword searching across all the indexed fields rather than providing access by querying information in specific fields. 20 The importance of vocabulary control and special indexes becomes obvious when searching across multiple databases in this manner. Keyword searching results in high recall and low precision. Since relevancy ranking is not an option with Z39.50, the results are not displayed in any kind of meaningful order. This means that the user may find this kind of searching only marginally better than searching the Web or looking for a needle in a haystack. 21 The Illinois Digital Library Project at the University of Illinois, Urbana-Champaign was one of six experimental prototypes funded by the Digital Library Initiative. 22 In the first phase, the project developed a testbed of full-text science articles from various publishers that could be searched as a single collection. Instead of Z39.50, the project staff designed a an interface, DELIVER, that allows patrons to search full-text documents that are in the SGML format; each document has its own article-level metadata. 23 The SMGL tags provided by the publishers were mapped to a standard metadata element set to support federated searching for specific tags such as Author, Title, and Subject. On a smaller scale, some commercial vendors provide federated searching over the aggregated databases that they offer through subscription. A recent survey compared the vendors according to the level that individual databases were mapped to a standard for indexing and vocabulary control to those who use machine-aided or automatic indexing for authority control or full-text searching with relevancy ranking. 24 The survey found that there is a big difference in how well the various vendors dealt with the differences in vocabulary control. Vocabularies By using standardized terms and phrases for identifying people, places, corporate bodies and concepts, it is possible to greatly improve retrieval of relevant information associated with a particular concept. Some databases provide access to the controlled terms, along with cross-references for variant forms or words that point the searcher to the preferred form. This optimizes the searching and retrieval of digital objects (bibliographic records, images, sound files, etc.). However, there is no universal authority file to which catalogers/indexers/users can consult. Each cataloging or indexing domain has developed its own thesauri or lists of terms that are designed to support the research needs of a particular community. Crosswalks have been used to migrate the data structure of a vocabulary from one format to another, 25 but only recently have there been projects to map the data content that actually populates that structure. When searching many databases at once, precision and relevancy become even more important. This is especially true if one is searching single-search query bibliographic records, records from citation databases, and full-text resources at the same time. Integrated authority control would significantly improve both retrieval and interoperability in searching disparate resources like these. 26 The Gale Group and Dow Jones Interactive have approached the problem of multiplesubject thesauri by creating a single thesaurus and mapping the controlled vocabulary from the individual databases to their in-house thesaurus. It is unclear to what extent the Mary Woodley, page 7 of 10

8 depth and coverage of the controlled terms in the individual databases is compromised by this merging. 27 Conclusion Crosswalks support conversion projects and semantic interoperability to enable searching across heterogeneous distributed databases. Inherently, there are limitations to crosswalks; there is rarely a one-to-one correspondence between the fields or data elements in different information systems. Mapping the structural components of metadata systems is only one part of the picture. Crosswalks of search commands and controlled terms or thesauri will further enhance searchers ability to retrieve the most precise search results. As the number and size of online resources increases, the ability to refine searches and to use the controlled vocabularies and thesauri will become increasingly important. Examples of other Crosswalks: Crosswalks from the Alexandria Metadata Schema to Other Schemas: ADL to FGDC; ADL to FGDC to USMARC to GILS Elements; FGDC to USMARC; USMARC to FGDC. Last viewed April 24, Network Development and MARC Standards Office. Dublin Core/MARC/GILS Crosswalk Last updated: 14 October Last viewed April 24, 2000 Metaform: Crosswalks, Crosscuts, & Mappings. State and University Library at Göttingen, Germany (SUB) Day, Michael. Metadata: Mapping between Metadata Formats International Federation of Library Associations and Institutions Metadata Resources 1 An excellent survey of the history and future of library automation can be found in Borgman, Christine. From Acting Locally to Thinking Globally: A Brief History of Library Automation, Library Quarterly. 67(3) (July 1997): There is no standard terminology for these discrete bits of information. In various systems they are referred to as: fields, labels, tabs, identifiers. St. Pierre, Margaret and William P. LaPlant, Jr. Issues in Crosswalking, Content Metadata Standards. NISO Standards. o.org/crsswalk.html. Last viewed April 24, Ibid. Mary Woodley, page 8 of 10

9 4 A good description of the problems that face converting museum data from one standard to another is in Dolen, Richard. The Nature of Museum Data Conversion, Spectra 26, 1 (Spring 1999): Home page for the organization is at: 6 Dublin Core Element Set, Version 1.0: Reference Description htm/. Also published as: Weibel, S.; Kunze, J.; Lagoze, C.; Wolf, M Dublin Core Metadata for Resource Discovery. IETF #2413. The Internet Society, September A list of the available qualifiers can be found on the Dublin Core Web Site. The work for formally approving the qualifiers are still in progress Consortium for the Computer Interchange of Museum Information (CIMI). Guide to Best Practice: Dublin Core (DC1.0=RFC2413). August 12, Kempe, Deborah. SCIPIO Art and Rare Books Catalog File: Perspective from a Valued User and Contributor. RLIN Focus. 40 (Oct. 1999). Last viewed April 24, Kempe, Deborah. Report and Recommendations of the SCIPIO Advisory Task Force. February, Last viewed April 24, The Library of Congress Name Authority File (NAF) is a union list of personal and corporate names (and titles) shared by the library community. The preferred forms of the names and cross-reference forms (variants) are established under strict guidelines established by the Anglo-American Cataloging Rules. 16 This table is based on two unpublished documents: Bales, Kathy, SCIPIO Conversion to ITPS, RLG document dated January 22, 1997; and Gunson, Alison, Migration of SCIPIO File form RLIN to IRIS, Getty Research Institute document, October 21, I also wish to thank Prima Casetta and Ani Matosian at the Getty Research Institute for providing details concerning the download of SCIPIO records into IRIS. 17 Huwe, Terence K. New Search Tools for Multidisciplinary Digital Libraries. Online v. 23 on. 2 (March 1999): The Boolean operators and, or, and not specify the relationship between search terms. 19 Truncation broadens the use of search terms by using symbols such as! or * to replace letters at the end of string for variant spellings. Wildcards use symbols to replace letters within a word. 20 SearchLight became available in January, Pharos is scheduled for public access in June, The UC site suggests that the patrons search the individual databases for precise search results and warn that the broad keyword search will produce extraneous and irrelevant results. Mary Woodley, page 9 of 10

10 22 Schatz, Bruce and Hsinchun Chen. Digital Libraries: Technological Advances and Social Impacts. Computer (Feb. 1999): Information concerning Phase 1 and 2 of the DLI can be found at its Web site 23 Schatz, Bruce. Et al. Federated Search of Scientific Literature. Computer (Feb 1999): Milstead, Jessica L. Cross-File Searching. Searcher 7 (1) (May 1999): Union List of Artists Names (ULAN) is also available for searching on the Web ( 26 Vellucci, Sherry L. Metadata and Authority Control. LRTS 44 (1) (Jan. 2000): Milstead (1999). Mary Woodley, page 10 of 10