Apache Tika for Enabling Metadata Interoperability

Transcription

1 Apache Tika for Enabling Metadata Interoperability Apache: Big Data Europe September 28 30, 2015 Budapest, Hungary Presented by Michael Starch (NASA JPL) and Nick Burch (Quan,cate) Proposed by Giuseppe Totaro (Sapienza University of Rome) and Chris Ma=mann (NASA JPL) Contact:

2 Summary What is Apache Tika Tika and Metadata Metadata Interoperability Tika for Enabling Metadata Interoperability Conclusion and Future Work 2

3 Apache Tika as the de facto babel fish for digital documents WHAT IS TIKA 3

4 What is Tika Java- based toolkit to detect and extract metadata and text from heterogeneous files Built from source code using Maven Provides a single Parser interface to wrap around third- party parsing libraries Enables recursive parsing Performs language detecuon and translauon Used either programmaucally or standalone 4

5 Supported Format HTML, XML, XHTML MicrosoZ Office document formats OpenDocument Format iworks document formats EPUB, PDF, RTF Compression and packaging formats Text formats Audio, Image and Video formats Mail formats 5

6 DetecUon Tika tries to idenufy the right file type Custom MIME types registry DetecUon methods File name Content-Type hints MAGIC bytes Character encodings Combined approaches 6

7 Extending Tika Parsers Add your MIME- Type (Uka- mimetypes.xml) <mime-type type="application/x-isatab-investigation"> <_comment>isa-tab Investigation file</_comment> <magic priority="50"> <match value="ontology SOURCE REFERENCE type="string" offset="0"/> </magic> <glob pattern="i_*.txt"/> </mime-type> Create your Parser class public class ISArchiveParser implements Parser public void parse() {...} } 7

8 New Features The last stable release is Tika 1.10 Some new features of the last releases: Upgraded to Java 7 (TIKA- 1536) ExtracUon of biomedical informauon relying on Apache ctakes (TIKA- 1645, TIKA- 1642) ProbabilisUc mimetype detecuon tika-batch module for directory to directory batch processing 8

9 ExtracUon of metadata with Apache Tika TIKA AND METADATA 9

10 What is Metadata Informally defined as data about data DescripUve, structural, administrauve, rights management, preservauon [NISO (2004)] E.g., Title, Author, CreaUon Date, Rights Every metadata schema may vary a lot: Naming (e.g., Description and Info) Correspondences (e.g., Creator and FirstName / LastName) RepresentaUon (e.g., and 2007/01/01 ) 10

11 Model PerspecUve of Metadata A Survey of Techniques for Achieving Metadata Interoperability 11 Universal Modelling Language Meta- Meta-Model M3 instance of Abstraction Levels Schema Definition Language Metadata Schema Meta-Model instance of Model M2 M1 instance of [HASLHOFER, KLAS (2010)] Meta data M0 Fig. 5. Metadata Apache: building Big Data blocks Europe from 2015 a model perspective 11

12 Tika and Metadata Tika enables metadata extracuon (if present) Tika maps metadata onto common, consistent key- value pairs in Metadata (Some) Metadata APIs org.apache.uka.metadata Metadata class: mulu- values metadata container TikaCoreProperties: core set of basic properues org.apache.uka.xmp XMPMetadata: conversion to XMP model 12

13 Solr s ExtracUngRequestHandler Apache Solr uses Tika to ingest binary and/or structured documents Solr's ExtractingRequestHandler uses Tika to upload binary files into Solr Input parameters (configurauon) fmap.<source_field>=<target_field> uprefix=<prefix> Performs only name mapping 13

14 Metadata Roadmap The six- point roadmap includes: Reorganize metadata keys internally Move XMP output to an extra XMP module of Tika Correct parsers where necessary Add support for structured data to metadata class Introduce versioning scheme for metadata mappings Introduce the ability for clients to define own mappings 14

15 Interoperability as prerequisite for uniform data access METADATA INTEROPERABILITY 15

16 Metadata Interoperability Prerequisite for uniform access to media objects Metadata interoperability is a qualita2ve property of metadata informa2on objects that enables systems and applica2ons to work with or use these objects across system boundaries. [HASLHOFER, KLAS (2010)] 16

17 Metadata HeterogeneiUes [HASLHOFER, KLAS (2010)] Predominat heterogeneiues have been originally idenufied by: [Sheth, Larson (1990)] [Ouksel, Sheth (1999)] [Wache (2003)] [Visser et al. (1997)] 17

18 Examples of Metadata HeterogeneiUes 18

19 Interoperability Techniques Model Agreement e.g., Standardized Metadata Schema Meta- Model Agreement e.g., Global Conceptual Model Model ReconciliaUon Language Mapping (M2) Schema Mapping (M1) Instance TransformaOon (M0) Metadata Mapping 19

20 Metadata Mapping Technique that subsumes: schema mapping instance transforma2on crosswalks funcuons S s Given a source schema and a target schema, each consisung of a set of schema elements e s S s and e t S t, M is a direcuonal relauonship between two sets of elements e s i S t t and e j S t. S t 20

21 Mapping RelaUonship m M m p P f F cardinality mapping expression instance transforma2on func2on Mapping expressions [Spaccapietra et al. (1992)]: Exclude Equivalent Include Overlap I ( s t e i ) I e j ( ) = ( ) I ( t e ) j I e i s I ( s e ) i I ( t e j ) I ( t e ) j I ( s e ) i I ( s e i ) I ( t e ) j I ( s e ) i I ( t e j ) I ( t e ) j I ( s e ) i 21

22 Elements of Metadata Mapping [HASLHOFER, KLAS (2010)] 22

23 Introduce the ability for clients to define their own mappings TIKA FOR ENABLING METADATA INTEROPERABILITY 23

24 Tika for Enabling Metadata Interoperability To be integrated into Tika (TIKA- 1691) as new component Based on the following improvements: MappedMetadata class Mapping uuliues (schema and instance) MetadataConfig class Maps the metadata to a mediated schema 24

25 MappedMetadata class Wrapper of Metadata class Decorates two methods of Metadata: get: maps metadata on geber side (default) set: maps metadata on seber side 25

26 UUliUes and ConfiguraUon Mapping Methods CrosswalkUtils class (schema) TransformationsUtils class (instance) Mapping ConfiguraUon MetadataConfig class works as well as TikaConfig (parse XML config file) Fine- grained configurauon Default configurauon in tika-metadata.xml 26

27 Example of MappedMetadata (1/2) 27

28 Example of MappedMetadata (2/2) 28

29 Future direcuons for Uka- metadata CONCLUSION AND FUTURE WORK 29

30 Conclusion Oka- metadata is a new component to enable metadata interoperability on client side Pros Highly configurable technique Fine- grained mapping Cons Configuring a new mapping from scratch may require much Ume Unknown metadata mapped only to dynamic fields 30

31 Future Work IntegraUon with the next releases of Tika Simplify configurauon and provide a complete sample with documentauon Support strategies for unknown metadata Return current mappings as graphical representauon (i.e., Hierarchical Edge Bundling) 31

32 Acknowledgements This work has been started by Giuseppe Totaro and Chris Mapmann at NASA JPL Thanks to Michael Starch who has presented this proposal Thanks to Nick Burch who is kindly supporung this idea Thanks to Tim Allison who is acuvely commenung Jira issue TIKA