CDL s OAI Harvest Architecture Development

Transcription

1 California Digital Library CDL s OAI Harvest Architecture Development Presentation Wednesday, Bill Landis, Metadata Coordinator, CDL

2 Overview OAI harvesting infrastructure planning and development thus far (Bill) Developing CDL capacity to harvest, transform/normalize, and ingest metadata-only digital objects into a CF repository Overview of results of Metadata clustering experiments and possible next steps (Dave) Possibilities for using clustering and classification as a means of enriching CDL digital objects (American West? Others?) with consistent metadata to enhance topical browsing access for end users Questions/discussion

3 CDL Common Framework Context Target for long-term management of American West metadata objects: Preservation Repository at CDL Repository based on OAIS Reference Model Records harvested to file system Feeder process produces METS object/sip Ingest process for Repository creates AIP, or an enriched/transformed AmWest metadata package Access services driven by indexes created from Repository METS objects

4 OAI Harvest Architecture OAI Provider OAI Provider U.Mich. OAI Harvester OAI Provider Harvest Tracking DB OAI Sets On File Server

5 Prototype Harvest Tracking DB UMich OAI harvester Stores Goals Especially like its ability to log harvest details (time, progress, completion, # of records harvested) -- automatically fed into Harvest Tracking DB Metadata about data providers/ institutions Information about who set up harvest for which project (important in CDL environment) Initial harvest dates and re-harvest frequency Feed information about an object s harvest into METS record creation process Feed information to CDL s automated scheduling software for future re-harvests Currently OAI specific/we ultimately envision this being folded into CDL Common Framework Tracking DB (along with tracking information for Preservation Repository ingest, Web crawling, etc.)

6 Prototype Harvest Tracking DB MySQL database Most information auto-populated during harvesting process Tables for Institution - Data provider Harvest Set - User phpmyadmin user interface Supports human-driven (as opposed to automated) interactions with data providers Supports human input of certain field values Public access site URL at Set level if this exists Set-specific or Repository-specific Rights URL when this is not supplied in OAI records or in Identify or ListSets responses -- in these cases CDL will add this human-input Rights URL from the Tracking DB during the course of METS-record creation

7 Input OAI data provider URL Select OAI verb to send

8 Choose sets for which to request records Edit information about set in tracking DB

9 In Edit mode, add contextual and other information for later use in normalizing/enriching record. * In this case LC provides the set s public URL in the set description, but not in each record, so we record it here to be added to our transformed metadata record (AIP). * We also add the generic rights URL for the set or institution, depending on which is available, for use in case individual records do not contain a dc.rights field. * Other fields contain data auto-populated by the Tracking DB)

10 Switching to the ListRecords OAI verb request invokes a harvest of selected, previously unharvested sets.

11 DB provides browsing access to: OAI verb responses stored in Tracking DB Previously harvested sets Newly harvested sets

12 Access to individual records through the Tracking DB is both to the harvested XML file and to the <identifier> URL s destination

13 Harvested XML record

14 Identifier URL s destination

15 Metadata Transformation (Feeder/Ingest) SIP SIP SIP Subset filter Original Record (SIP) Enhanced Record (AIP) OAI Feeder Process (creates METS objects) Context DateTx Thmb Tx Context Information Date/Era Normalization Identifier validator/thumbnail generator GeoTx Geographic place names GenreTx Type/Genre OAI Sets Harvest Tracking DB TitleTx Title generator Future transforms? CDL Common Framework (CF) Repository Object

16 Repository SIP SIP SIP Feeder Process DateTx GeoTx TypeTx OAI Sets Harvest Tracking DB TitleTx Repo Clustering Index User Interface Access Services

17 Feeder Process for OAI Records METS record creation Successfully wrapped and ingested approximately 100,000 metadata-only OAI objects during development/testing in March New CDL Repository service: Catalog added to previously existing Preserve Untouched copy of as-harvested metadata becomes SIP Information from Harvest Tracking DB parked in METS record ( Context info on previous slide) for downstream use Default rights URL for set or repository as backup

18 Repository Ingest Process Generic ingest profiles Hope: generic profile will be able to take care of 90+% of records harvested Reality: we ll see Profiles essentially an ordered list of calls to CDL CF utilities for things like Normalization of specific fields Enrichment activities on specific fields Subsetting and removing out-of-scope content for a given collection Special profiles for specific sets We hope this is the rare exception, not the rule Ex: Only content date in set records are in <identifier> string!

19 Ingest Utilities Subsetting and disposing of out-of-scope content Improving best practices in OAI data provider community will hopefully make this easier as time goes by, both at set and record levels Software-assisted human analysis of OAI sets/records seems the best way to develop subsetting algorithms, probably tied to specific OAI service provider projects We may need to do this in a human-mediated step post Harvest, but pre Ingest

20 Ingest Utilities Date Normalization Coding completed, QA plan in development Stats from last test run: 359,830 records processed, found and normalized at least one date in 279,337 records (76.4%) using <date>, <title>, & <description> fields Process Analysis of 360,000 records Roy s web-based analysis tools Spotfire DecisionSite ( - very large datasets problematic perl hacking courtesy of Mike Information derived from clustering experiments (probably more useful for geographic and genre information than for dates)

21 Ingest Utilities Date Normalization process (cont.) Developing algorithm Steps for normalizing date patterns extracted from <date> fields Steps for extracting date patterns from other fields if no <date> field or if <date> field contains some version of undated/unknown/n.d./s.d., etc. Currently limiting this to <title> and <description> fields Steps for writing out normalized dates in CDL qualified Dublin Core fields in transformed AIP descriptive metadata Documentation, QA testing

22 Ingest Utilities Other transformations needed to support envisioned interface we ll be testing on target American West user populations beginning in 2006 ( Evaluation year for AmWest): Work on these ongoing during remainder of 2005, into 2006 Date normalization utility work is prototype for process for other utilities Prototype vs. production approaches for CDL CF development?

23 Ingest Utilities Higher priority to support evaluation activities: Identifier validator/thumbnail generator All of our user research indicates that end users want thumbnails in results lists most OAI records don t provide such links We anticipate that this utility will test for validity of links and destination filetypes, grab image files, make low-resolution thumbnail for object (algorithm to be determined), store pointer in METS record Geographical location terms Analysis of 360K records underway Currently planning to transform to TGN hierarchical terms (CDL licensing of TGN from Getty is in process) Will need to prototype some kind of basic, back-end Thesaurus Manager utility with which normalization utility can interact Derived from content of <coverage> and other candidate DC fields

24 Ingest Utilities Lower priority (can start evaluation activities and work these in during 2006) Genre terms Anticipating 2 hierarchical levels First subset of DCMI Type Vocabulary Second subset of AAT derived from analysis of current pool of harvested records Will also interact with a back-end Thesaurus Manager Derived from <type> and other candidate DC fields Title extraction utility When a record has no <title> field, we ll have algorithm for extracting approx. first 10 words of <description> field as a supplied title Approx. 800 of If no <description field> we ll insert an explicit <title>[no Title]</title> field for use in results lists and citations

25 Ingest Utilities Topic terms High priority to support American West evaluation activities beginning in November 2005 Several approaches to consider; more in Dave s talk Important for American West evaluation activities Feedback from usability testing of hierarchical, faceted browsing approach in American West interface will help CDL assess need for further development of CF support for clustering and classification activities Testing of usefulness of 23 High-level Topics Testing 23 topics locally yielded a 15-topic list Felicia/Jane designed online survey that was up for 2 weeks Bill solicited survey takers from K-12 teacher and historian listservs 194 responses Stay tuned for analysis