Copying Archives. Ngoni Munyaradzi (MNYNGO001)

Transcription

1 Copying Archives Ngoni Munyaradzi (MNYNGO001) Abstract This paper focuses on the problem of trying to define a common exchange interface. That will be used to implement repository-to-repository transfers. The significance of addressing the issue is that it provides flexibility of manipulating the digital content stored in repositories in the manner archivists choose. Repository-to-repository transfers can then be used to implement preservation and migration of digital content. The paper discusses the current technologies and related work that has been done so far. Key Words: metadata, interoperability, Digital Libraries Introduction and Motivation The Copying Archives project is about copying digital content to and from Digital Libraries Systems[2]. Digital Library Systems store digital content of various forms e.g. journals or video content. To achieve replication of digital content between digital libraries, a common exchange interface has to be defined. The common exchange interface will allow for the heterogeneous software repositories to be interoperable. The project will focus mostly on implementing interoperability between the LOCKSS [13, 14], DSpace [15] and EPrints [5] software repositories. There are other popular software repositories in use in the Digital Libraries community, like Greenstone [18] and Fedora, to name a few. For the purposes of this project, LOCKSS, DSpace and EPrints will be implemented. In the next section, current technologies in Digital Libraries that are used in the Copying Archives project will be discussed. Summary of Technologies LOCKSS (Lots of Copies Keep Stuff Safe) [13, 14] is a digital preservation system that collects content from a target Web site using a crawler and keeps this data in the format provided by the publisher. The system continually compares the content it has stored with other LOCKSS boxes, and repairs any differences; this is achieved by a voting system [8] it implements. The system acts as a Web proxy, providing browsers with access to the publisher s content. The system also supports content migration in the case that the format has become obsolete. LOCKSS is an OAIS compliant system [7]. In a LOCKSS network, there are multiple caches and these are decentralised; each cache contains the same content as the other caches on the network. The ability for the same content to be stored in multiple machines ensures that preservation of digital content is achieved, in the case that a cache fails. Page 1

2 DSpace [15] is an open source software repository that adheres to OAI and OpenURL compliancy. It allows users to capture and describe digital works using a custom workflow process. DSpace ensures long-term preserving of data and distribution of institution s digital works over the Web. The DSpace system provides a way to manage these research materials and publications [15]. It is mostly used in institutions to store their scholarly articles, but its use can be extended to multiple disciplines. Dspace archives a large variety of types of content such as articles, datasets, images, audio and video files, working papers and drawings. Dspace supports the Dublin Core metadata format [17]. Eprints [3] is a tool that is used to manage the archiving or research in the form of books, posters, or scholarly articles. It purpose is not to provide a long-term archiving solution that ensures that material will be readable and accessible through technology changes. Instead it gives institutions a means to collect, store and provide Web access to material. It is a free open source package, that is Open Archives Initiative (OAI)-compliant, which makes it accessible to cross-archive searching. Eprints comes with a user interface that can be customized. It also supports any metadata format. Preservation Digital preservation is the management of information over time. The set of processes and activities that ensure continued access to information and all kinds of records, scientific and cultural heritage. There has been little effort put into addressing the need for digital preservation [16]. Some people are arguing that this is an issue that should not be prioritised at the moment. The argument is that any system that has OAI compliance implements preservation. Digital preservation is important because software systems are susceptible to failure and becoming obsolete, hence this needs to be addressed. The Open Archival Information System [7] reference model was defined as a standard for digital preservation. The reference model provides a framework for understanding and increased awareness of archival concepts. The LOCKSS system was designed to ensure preservation of digital content. The Victorian Electronic Record Strategy (VERS) [16], has been developed to practically solve the problems of ensuring digital preservation. VERS designs a digital format, that preserves information indefinitely. The project has identified what it views might be keys to long term preservation of digital information. Encapsulation storing the information in a single location, having been enclosed with descriptive metadata. Self documentation being able to understand the information without needing references to other documentation. Self sufficiency minimising dependency of information. Content documentation descriptive information stated to allow other users to understand usability of content. Standardisation the digital information should be stored in standard formats e.g. METS [3], DIDL [1], MPEG [1]. Page 2

3 It is important to note that the LOCKSS system addresses the issue of preservation comprehensively, through the voting and polling system [9] it implements across the LOCKSS network. Hence, integrating the functionality of the former, DSpace and EPrints, will hopefully add to the current preservation techniques [16]. The next section discusses the role of preservation metadata. Metadata PREMIS[19] has done research into the use of metadata for preservation, Preservation metadata is defined as the information a repository uses to support the digital preservation process. PREMIS focuses on metadata supporting the functions of maintaining viability, renderability, understandability and authenticity. The adoption rate of PREMIS is proved to be generally low [19]. PREMIS has a comprehensive metadata framework [19] description in their data dictionary. PREMIS addresses the issues of implementation, sustainability (through identifying the core metadata elements), metadata creation and capture (human agency vs. automatic capture) and interoperability (support reuse of existing metadata). Interoperability In the context of the Copying Archives project, interoperability is defined as being able to export and ingest digital content with LOCKSS, DSpace and EPrints. The exchanged information must remain in a valid format once ingested in one of the software repositories. Validity of data is important because this ensures authenticity, and that all data has been copied across. Efforts have been made to make heterogeneous repositories interoperable, and one such project is the Open Archives Initiative - Protocol for Metadata Harvesting (OAI-PMH)[6]. This provides an application-independent interoperability framework based on metadata harvesting. A possible way to achieving transfer of digital content in heterogeneous repositories would be through the OAI-PMH framework. It requires use of the Dublin Core [17] metadata standard, which allows flexibility in defining the required fields in metadata content. The next section discusses possible issues of data integrity in wanting to design the common exchange interface. Issues in Data Integration In wanting to develop a common exchange interface between heterogeneous repositories, particular issues need to be considered and ways of solving these established: Heterogeneous infrastructure [12] there are a number of heterogeneous repositories and most of these store data in different formats. To extract data from the repositories, there is a need to manipulate export tool functionality of each repository. Insufficient data quality [12] - every repository is designed for a specific purpose, and this requires a certain level of detail of data stored and accuracy. Consistency in the Page 3

4 data is hard to obtain as the various platforms require different levels of metadata detail. Semantic gaps [12] contradictory fields need to be resolved after data integration of separate repositories has be done. Semantic mismatch is the cause of contradictory entries, as the fields for data are used for different purposes in each individual repository. Language support [11] the language supported in searching or retrieving data from the repository must be the same for all Digital Library Systems. Copyright / rights management [11] - particular information may be protected under copyright law and permission has to be obtained to transfer it from publishers. Naming and identifier [11] - Digital Library Systems name objects in different ways and unique identifiers are important to avoid naming conflicts. Related Research Efforts have been made in the digital libraries community to achieve and interoperability amongst heterogeneous repositories, for example a transfer from Greenstone [18] to DSpace [14] and vice versa. The two repositories share some similarity [18] and have differences [18]- the differences arise because the intended goal of both systems differ. Both these architectures implement the OAI-PMH [6] protocol; this would give an initial perspective that transfer of digital library content could be achieved through this common interoperability framework. However, through testing it has been shown that there are challenges associated with using the framework [6]; the level of integration would only be broad and not deep. Transfer of metadata would be achieved successfully but not the actual object itself due the object identifier element. This is because the OAI-PMH protocol does not support digital object migration. Integration can possibly be done through the METS [3] standard. METS would offer a deeper level of integration via its combined metadata and document container approach. Both Greenstone and DSpace have the capability to export through this format. The METS standard is flexible, but has a disadvantage that different systems might implement structures that might not map to one another. DSpace supports a hierarchical form of metadata while Greenstone metadata is flat, and to achieve integration these differences need to be reconciled. StoneD [18] implements transfer of objects and metadata through import/export, which allows greater level of integration than the previous two methods mentioned. This allows Greenstone to access documents in DSpace at the database level. Another possible alternative to accessing data at the database level is to do it at a service level. Page 4

5 The Towards Interoperable Preservation Repositories (TIPR) [8] projects aims to do testing, refining of transfer mechanisms, and repository to repository transfer. TIRP uses 3 basic Information Packages defined in the OAIS standard to implement repository-torepository transfer. The 3 information packages [8] are, Archival Information Package (AIP), Submission Information Package (SIP)and Dissemination Information Package (DIP). The details of the four projects on the TIPR experiments were published in a special edition of D-Lib Magazine in Results obtained showed that there are many different ways in which the exchange of information packages could fail [8]. TIPR noted that a key requirement for any transfer format is that it carries rich technical and historical information supplied by the source repository. Not all repositories will be able to interpret this information, but the originating repository should communicate the data. METS is the primary schema, and other schemas are used within METS to extend it. Although METS was used in the four projects, no repository could take a package produced by another project and ingest it without substantial transformation. TIPR have defined a Repository exchange Package (RXP) [8], for exchanging information. The RXP includes both package-level and representation-level PREMIS information. A DIP[8] from a repository is transformed into a RXP, which can be ingested by any other repositories. The research done by the two projects StoneD and TIPR serve as a starting point of identifying which standard and techniques are best in implementing repository-torepository transfers. Summary of competing methods/critical Comparison In every digital library system there is an overarching framework within which everything is held. Two standards have emerged, which aim to fulfil this function: the Metadata Encoding and Transmission Standard (METS) [3] and Digital Item Declaration Language (DIDL) [1]. The two standards have frameworks within which descriptive, administrative and structural metadata are defined. Both have mechanisms for recording data of the individual files that define the digital library object, plus methods of recording information on how the data should be rendered when the user receives them e.g. detailing information about what software should be used on the object. Both provide mechanisms for recording the internal structure of a digital object, in a nested hierarchy form, so that it makes sense to the user. The way in which these two standards implement the functions is different. METS is arranged according to the types of its constituent metadata (descriptive, administrative and structural), each of which is located in different sections of its overall structure. DIDL collates all types for a given component together. For example, an image file will have its descriptive and administrative metadata located together. METS employs a separate structural map to act as the centre for links to the metadata, while DIDL embeds everything into a single hierarchy, representing the structure of the digital object. The METS standard proves to be the most flexible to use, hence its wide adoption. The above-mentioned content packaging and preservation techniques, are all defined in XML, schemas and DTDs. Page 5

6 Conclusion A number of the papers discussed mostly point out that research into repository-torepository transfer is still ongoing. As mentioned in the discussed papers, there has been a growing acknowledgment for the need to design repositories that support preservation, to avoid data loss. Progress has been made in identifying some of the important transmission packages that can be used e.g. VERS, METS and PREMIS are being used together. References 1. Bekaert J, Hochstenbach P, and Van de Sompel H. Using MPEG-21 DIDL to represent complex digital objects in the los alamos national laboratory digital library. D-Lib Magazine 2003; 9: Borgman CL. What are digital libraries? competing visions. Information processing and management 1999; 35: Cundiff MV. An introduction to the metadata encoding and transmission standard (METS). Library Hi Tech 2004; 22: Gartner R. Metadata for digital libraries: State of the art and future directions. JISC TechWatch Report 2008;. 5. Gutteridge C. GNU EPrints 2 overview. 2002;. 6. Lagoze C, Van de Sompel H. The open archives initiative: Building a low-barrier interoperability framework. 2001; Lavoie BF. The open archival information system reference model: Introductory guide. Microform & imaging review 2004; 33: Magazine DL. Repository to repository transfer of enriched archival information packages. D-Lib Magazine 2008; 14: Maniatis P, Rosenthal DSH, Roussopoulos M, Baker M, Giuli T, and Muliadi Y. Preserving peer replicas by rate-limited sampled voting. ACM SIGOPS Operating Systems Review 2003; 37: Paepcke A, Chang CCK, Winograd T, and García-Molina H. Interoperability for digital libraries worldwide. Commun ACM 1998; 41: Pinfield S, James H. The digital preservation of e-prints. D-Lib Magazine 2003; 9: Ramler R, Wolfmaier K. Issues and effort in integrating data from heterogeneous software repositories and corporate databases. 2008; Reich V, Rosenthal DSH. LOCKSS: A permanent web publishing and access system. Sun Microsystems Laboratories The First Ten Years 2001;. 14. Reich V, Rosenthal DSH. Lockss (lots of copies keep stuff safe). New Review of Academic Librarianship 2000; 6: Smith MK, Barton M, Bass M, Branschofsky M, McClellan G, Stuve D, Tansley R, and Walker JH. An open source dynamic digital repository. D-Lib Magazine 2003; 9: Page 6

7 16. Waugh A, Wilkinson R, Hills B, and Dell'Oro J. Preserving digital information forever. 2000; Weibel S. The dublin core: A simple content description model for electronic resources. Bulletin of the American Society for Information Science and Technology 2005; 24: Witten IH, Bainbridge D, Tansley R, Huang CY, Don K, and Hamilton NZ. A bridge between greenstone and DSpace. D-Lib Magazine 2005; 11: Woodyard-Robinson D, Ltd WRH. Implementing the PREMIS data dictionary: A survey of approaches. 2007;. Page 7