Metadata Overview. November, 1999 c Grace Agnew

Transcription

1 Metadata: Data that describes data Data that manages data Structured data about data Pure metadata has meaning only in relation to the primary data that is being described.

2 Types ofmetadata:! Structural Describes the physical and logical attributes of the object, related to creation, transport, storage and display; Describes the hardware and software used to create the object; Describes the hardware, software and bandwidth needed to transport and display the object. May be machine-readable, human readable or both. May be part of digital object header (ex: TWAIN)

3 ! Provenance/Ingest Metadata: Admission ticket to the Archive or Data Repository. Acknowledges the rules of entry and identifies the object for positioning within the Archive. Best if intrinsic in the object, e.g. In the Header. Identifies the owner/creator of the metadata. Identifies the owner/creator of the digital asset. Provides date created, permanence of asset; updates and modifications to asset. May push asset to users when content changes.

4 ! Rights & Access: Provides requirements for access, display and download/storage of asset. Should integrate with organization s access and authorization system, e.g. Reference/hyperlink to Digital certificate authority Indicate User restrictions (may reference attribute on certificate authority s user attribute server Support multilayered access: download only vs. store; free vs. fee; asset versions (high res. Vs. low res.)

5 ! Descriptive Should uniquely identify an asset through: Physical description (overlap with structural metadata) Publication/Creation information (overlap with ingest metadata) Should describe the information content in subject and free-text fields to identify and select the asset in response query from a search engine.

6 ! Linking Metadata Persistent Links: Metadata record and the described asset. All physical instantiations of the asset. Registries for metadata schemas used to provide a metaschema to describe the object. Security system for access and authorization and/or link to intermediary access page Considerable overlap with other metadata types

7 Mining Web Assets: Current Practice A query is sent to a proprietary search engine, or a metasearch engine which queries many engines. Benefits: Ubiquitous and free; competition results in better precision and coverage Drawbacks: Access for assets only, not long-term management; Ephemeral metadata; Asset creator has no control over description and access.

8 Standards are Developed to: " Create durable, persistent metadata records that precisely define the asset so that exactly-relevant assets are identified and retrieved in response to a query. " Allow a community to describe and manage shared assets " Allow the metadata records from many schemas with differing levels of complexity to interoperate for data discovery. " Enable machine-intervention for automatic interpretation of metadata and data discovery, particularly among disparate search and retrieval platforms

9 ISO Joint Standard of the ISO (International Organization for Standardization) and IEC (International Electrotechnical Commission) to provide a robust framework for defining data elements in an unambiguous and persistent manner within user committees. Also provides a framework for creating and maintaining metadata registries to store and maintain data element definitions. NCITS L8 Draft Standards available at the following website:

10 RelevantMetadata Standards: Dublin Core Element Set V. 1.1 (IETF Recommendation) - Flexible lowest common denominator standard with 15 optional, repeatable fields; - XML and HTML based - integrates completely with assets that live on the web or are accessed via the web and live in an attached database May be intrinsic or separate from the asset described; - Automated tools for generating/validating Dublin Core are freely available, e.g. DC.dot: - HTML example:

11 Content Intellectual Property Instantiation Title Creator Date Subject Publisher Type Description Contributor Format Source Rights Identifier Language Relation Coverage From Description of Dublin Core Elements

12 Dublin Core Drawbacks Too Flexible and Simple for complex, sophisticated collections; Elements lack standardized use and precision. Different communities are developing extensions to specify and categorize the elements. Approved extensions are available but slow to appear. Some elements (rights, coverage) are ambiguous in their application Intended for web objects that are textual or primarily textual. Does not provide for: components (e.g. a video shot ; sequential media (audio and video, slide shows); synchronized media (e.g. video with closed caption transcription)

13 Result: Every Community Creates Their Own Metadata Archives: EAD (Encoded Archival Description) Government: GILS (Government or Global Information Locator System) IMS: TEI: Dublin Core Flavors Instructional Metadata System Text Encoding Initiative - books and humanities; TEIH (TEI Header used for metadata description EdNA CIMI Guide to Best Practice: Dublin Core. Available as PDF from

14 MARC Machine-readable cataloging: most library catalogs worldwide. MPEG-7 Digital Audio, Video and Still Image files. (In development. Committee draft due October 2000)

15 M PEG-7: Intended to describe audiovisual information regardless of storage, coding, display, medium of technology--will include analog and digital media and combinations of media formats Will Standardize: * Core set of Descriptors (D) * Description Schemes (codified structures of Descriptors-- definition, constraints, relationships among Descriptors) (DS) * Language defining Description Schemes and Descriptors

16 MPEG-7 Structural Model Jane Hunter. MPEG-7: Behind the Scenes in D-Lib Magazine September, 1999 (v. 5, no. 9): 6)

17 Possible MPEG7 schema incorporating DC <DC:Type>Image.Moving.TV.News.sequence.scene</DC:Type> <DC:Description.text> Footage of Grenade Attack </DC:Description.text> <DC:Description.transcript> Sam Rainsy knows the violence of political life in Cambodia. Four months ago, 16 of his supporters were killed in a grenade attack in Phom Penh. </DC:Description.transcript> <DC:Format.Length>10seconds</DC:Format.Length> <DC:Coverage.t.min DC.Scheme= SMPTE >19:31:57;1</DC:Coverage.t.min> <DC:Coverage.t.maxDC Scheme= SMPTE >19:32:07;1</DC:Coverage.t.max> From: Jane Hunter and Renato Iannella. The Application of Metadata Standards to Video Indexing

18 Beyond the Metadata Schema: Access to Information: " Information stored and managed within your organization (possibly under different metadata schema) " Information stored and managed by outside organizations

19 Books and web sites written by Grace Agnew Author: Agnew, Grace Parameter mapping: DC.Creator, DC.Contributor Metadatabase - Dublin Core Record 1 DC.Creator Grace Agnew Record 70 DC.Contributor. Grace Agnew Result Set: AGNEW, GRACE 1999 AGNEW, GRACE

20 Books and web sites written by Grace Agnew SEARCH ENGINE 1 Metadata schem a 1 SEARCH ENGINE 2 Metadata schem a 2 Author: Agnew, Grace Parameter mapping: DC.Creator, DC.Contributor Author: Agnew, Grace Parameter mapping: 100, 700

21 Z39.50 Information Retrieval (Z39.50): Application Service Definition and Protocol Specification Enables a client to interact with multiple servers, employing different search engines and different data element formats and definitions, to search databases and retrieve the records that result from the search

22 Z39.50 " Initiates a session between client and server " Executes a query from the client against one or more databases on the server " Creates a result set consisting of records that match the query on one or more query attributes (access points)

23 Z39.50 " Returns a report on the number of records matching the search " Returns records--individual records selected by the client--in a format selected by the client " Primary formats returned: MARC, SUTRS, extending to SQL, Dublin Core, other schema

24 Z39.50 Version 3 Extends the capabilities of the standard to include: Boolean and proximity searching Extended services, including saved queries to be periodically re-executed ( SDI ) Explain facility to allow client to solicit information about the server and dynamically reconfigure itself.

25 Z39.50 Profiles for User Groups: LOC: Access to Digital Collections LOC: Access to Digital Library Objects CIMI: Companion profile for museum digital collections and objects GEO: Geospatial Datasets Z+SQL: extension to the SQL query language

26 Z Limitations Requires client software and Z39.50-enabled server software (which requires Z39.50 aware search engine) Most commercial C/S Products have not implemented the explain feature in version 3 Requires human collaboration for implementation, particularly at the profile level Limited primarily to features provided by commercial servers and clients

27 M etadata Registries: Dynamic specification, maintenance and description of metadatabase structures: unambiguous definition of data structures unambiguous definition and description of relationships between data structures, behaviors of data structures, integrity constraints on the contents of data structures. semantics (meaning in context) and structure definition

28 M etadata Registries Links/Hooks into subordinate registries used to define data content within a metadata element Mapping of data structures between registries Should be both eye-readable and able to be interpreted by computer programs for seamless, unambiguous discovery, query and display across disparate database and search engine structures and to enable intelligent query agents, advanced data mining, etc.

29 M etadata Registries Collaborative Effort of the Joint Technical Committee 1 (JTC1) of the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) Open Forum on Metadata Registries: events/openforum/index.htm

30 M etadata Registries REGGIE -Java Applet that dynamically creates metadata according to available online registries; Allows you to enter your own registry, describing, characterizing and constraining all the elements in the set. UK/Australia joint effort

31 Anything by Grace Agnew? Metadatabase Scheme = DC <URL of Registry> Dublin Core. Author defined as: Creator, Contributor

32 R esource Description Fram ew ork W3C Resource Description Framework (RDF) Model and Syntax Specification (22 February 1999): Provide robust application of metadata in the web environment: Model for unambiguous, schema-independent description of resources. Key Concepts: Resource: Any object uniquely identifiable by a URI (uniform resource identifier) Property-type: Property associated with a resource. Value: Associated with a property type--may be atomic (a string) or another resource, creating a new hierarchy)

33 RDF Property types express the relationships of values associated with resources: Famous Example The Author of Metadata Overview is Grace Agnew M etadata O verview Author Grace Agnew o Property Value Type R esource

34 RDF Enables interoperability among metadata schemes, including the modular use of multiple schemes within a metadata record utilizing the XML namespace facility; Adds machine-interpretable semantics to the encoding, exchange and reuse of structured metadata; Enables automatic negotiation between search engine, metadata record, and metadata registry for powerful, flexible search and retrieval independent of server and client search and retrieval infrastructures (or, at least, it will!)

35 XM L Extensible Markup Language, a subset of SGML (Standard Generalized Markup Language) provides the ability to define elements within a web document, provides the ability to describe objects as well as documents; provides multidirectional linking. XML display is governed by an attached style document, formulated in CSS (Cascading Style Sheet) or XSL (Extensible Style Language) to provide rules for display. Styles can be applied to single elements as well as to the entire document. More than one style sheet or style document can be provided for a document or element, with precedence rules governing the given display.

36 DTD The Document Type Declaration provides a formally defined structure, vocabulary and syntax for an XML document type. Documents are validated against a DTD to insure nested structure and semantic constraints are followed to insure consistent meaning across documents. DCD A semantic superset of XML DTDs--intended to be conformant with the RDF Model and Syntax Specification. Describes an XML vocabulary for schemas--for specifying object classes. Based on elements (RDF property types) and attributes Supports RDF vocabulary and constructs.

37 SOX Schema for Object-Oriented XML Alternative to DTD for validating XML documents. Supports scalar (numeric) datatypes, enumerated datatypes (values enumeration) and format datatypes. An expanded namespace facility supports objects from any identifiable namespace to be used to build the document.

38 Role of the Database: A database that can be parsed and reported to a validated XML metadata format, as well as other metadata syntaxes, provides a robust space for metadata development. Also reports to any XML Document type and hooks into applications via APIs, to support unique user needs ORACLE DATABASE COLLABORATIVE RESEARCH SPACE MARC-BASED CATALOG PERSONAL RESEARCH SPACE SUBJECT- SPECIFIC WEB RESEARCH TOOL WEB-BASED COURSEW ARE APPLICATION

39 LastStep:Data Retrieval Data storage, access and delivery architecture: open, standards-based, hardware and software independent, providing users across platforms with common, consistent interface and underlying storage structure for efficient retrieval, display, storage and use of digital information Data architecture should support a well-defined, widely available security system to validate authenticity of users and provide data for a variety of uses according to a scalable authorization hierarchy

40 LastStep:Data Retrieval Data architecture should support data as objects for scalable, extensible access, with sophisticated and flexible support for object relationships, particularly to support different physical instantiations of identical data, e.g. digital video object as D1, MPEG1, Quicktime, etc. CORBA Common Object Request Broker Architecture - emerging architecture for open distributed object computing. Intended to provide transparent access to applications and databases, regardless of the hardware and software infrastructure at each end of the transaction

41 Putting ItA ltogether: A DigitalArchive Architecture Reference Model for Open Archival Information Systems (OAIS), Developed by a US ISO archiving group under ISO TC20/SC13 and the Consultative Committee for Space Data Systems (CCSDS). This model has recently been released for formal ISO and CCSDS review. An electronic version of the OAIS Reference Model can be found at http//ssdoo.gsfc.nasa.gov/nost/isoas/ref_model.html

42 Reference Model for Open Archival Information Systems (OAIS) EXTERNAL DATA FLO W DIAGRAM