Overview. What is Information Retrieval? Classic IR: Some basics Link analysis & Crawlers Semantic Web Structured Information Extraction/Wrapping

Transcription

1 Overview What is Information Retrieval? Classic IR: Some basics Link analysis & Crawlers Semantic Web Structured Information Extraction/Wrapping Hidir Aras, Digitale Medien 1

2 Agenda (agreed so far) 08.4: General Introduction part1 and administrative things 15.4: Introduction part : Assignment of Task1, Introduction part : Introduction part4: Crawling & Semantic Web 06.5: Task1: short presentations, Paper 1 (3 x min) 13.5: Topic: Information Extraction from the Web, Assignment of Task2 20.5: Short introduction into scientific paper writing 27.5: Task2: short presentation, Paper 2 (3 x min) 03.6: Open discussions & free topic (LSA/LSI?), Approval of papers for final presentation 10.6: Open discussions & free topic (Robert: NLP topics) 17.6: Task3: Final presentation, Paper A + B (3 x min) 24.6: Task4: Submission of the merged scientific Short Paper (A+B) 01.7: Free topic 08.7: Free topic Task 1 and 2 mainly for discussions, not grading. Task 3 and 4 will be used for grading. Hidir Aras, Digitale Medien 2

3 Information Extraction Information Extraction (IE) sub-discipline of IR methods to transform unstructured text or parts of semi-structured documents into a structured representation Hidir Aras, Digitale Medien 3

4 Web IE Extract pieces of structured or semi-structured data from web documents Wrapper Induction (supervised) vs. Automatic Data Extraction (unsupervised) Other approaches (Ontology-based, Instancebased extraction etc) Hidir Aras, Digitale Medien 4

5 Examples of Web Content Tabular data Examples: - Match results and reports - facts etc. Mannschaft Dänemark Senegal Uruguay Rang P g u v T T Pkt Frankreich News with unstructured and structured parts Example: - sports news, newsticker Fußball-Weltmeister Brasilien "Wir sind stark, aber nicht unschlagbar." Weltmeister Brasilien hat sich für das Spiel gegen die deutsche Mannschaft am Mittwoch schon mal warm geschossen: In der WM- Qualifikation wurde Boliven mühelos mit 3:1 besiegt. mehr...( ) <soccernews> <keywords> <keyword>fussball</keyword> <keyword>weltmeister</keyword> <keyword>brasilien</keyword> </keywords> <title>"wir sind stark, aber nicht unschlagbar."</title> <text> Weltmeister[WorldCup] [winner] Brasilien [Team] hat sich für... </text> <morelink> </morelink> </soccernews> Hidir Aras, Digitale Medien 5

6 Web IE : - Timeline * MUC (Message Understanding Conferences): Analyzing free text, identifying events of a specified type, and filling a data base template with information about each such events. Hidir Aras, Digitale Medien 6

7 IR vs IE Input Output Hidir Aras, Digitale Medien 7

8 Web Information Extraction Source: Hidir Aras, Digitale Medien 8

9 Example (1): IE As a task: Filling slots in a database from sub-segments of documents. October 14, 2002, 4:00 a.m. PT For years, Microsoft Corporation CEO Bill Gates railed against the economic philosophy of open-source software with Orwellian fervor, denouncing its communal licensing as a "cancer" that stifled technological innovation. Today, Microsoft claims to "love" the opensource concept, by which software code is made public to encourage improvement and development by outside programmers. Gates himself says Microsoft will gladly disclose its crown jewels--the coveted code behind the Windows operating system--to select customers. "We can be open source. We love the concept of shared source," said Bill Veghte, a Microsoft VP. "That's a super-important shift for us in terms of code access. Richard Stallman, founder of the Free Software Foundation, countered saying IE NAME TITLE ORGANIZATION Bill Gates CEO Microsoft Bill Veghte VP Microsoft Richard Stallman founder Free Soft.. Hidir Aras, Digitale Medien 9

10 Example (2): Web-IE/Wrapping <table border= 3"> <tr class= Book > <td>a Brief History of Time</td><a>link1</a> <td>eur 9.45 </td><a>link2</a> <td>s.hawking</td> Instance </tr> <tr> <td>joghurt</td> <td>1.24</td> </tr> </table> Class = Book : - title = A Brief - price = writer = S. Hawking Hidir Aras, Digitale Medien 10

11 The Wrapper Generation Problem Given a web page P containing a set of implicit objects Determine a mapping W that populates a data repository R with objects in P The mapping W must also be capable of recognizing and extracting data from other page P similar to P term similar used here in a very empirical sense, meaning pages created by the same web-script or service - Consequently, a wrapper is a program that executes the mapping W Hidir Aras, Digitale Medien 11

12 Example wrapper (attributed grammars) Example (attributed grammars) on HTML or other markup data: rule mytext is ( <.* > text &=.* )* end alternative iterator character matching optional repetition concatenate text inside markup match all, i.e. realizes skip all characters Hidir Aras, Digitale Medien 12

13 Definitions Definition 1: - A syntactic wrapper is a function W:P T. Given a web page P, it gives back a tuple with the information of interests. Definition 2: - Let L be an ontology, a semantic tuple is the result of properly associate? the information in the tuple with concepts defined using L. Definition 3: - A semantic wrapper is a function Ws: P Ts. Given a web Page P, it returns a semantic tuple with the information of interest. Hidir Aras, Digitale Medien 13

14 A (Semantic) Wrapper Architecture XSD template Template Generator SEMANTIC WRAPPER SW Ontology (SportEventOntology) retrieve data Data Extractor extract data XML Semantic Translator induce analyze wrapper generator <rdf:rdf> </rdf:rdf> insert / update Query (RDF-QL) Manager select-query RDF repository Instances of the SmartWebOntology Hidir Aras, Digitale Medien 14

15 Information Extraction & Machine Learning Key idea: learn a procedure (wrapper) that extracts text from a document similiar to previously given examples (text parts) General Questions: learn one wrapper for one page? learn one wrapper for several pages? positive and negative examples? learnability? choose which representation? which learning method? Depends on the task [E.M. Gold:67] in general we are not able to learn Regular Languages only from positive examples. But if we restrict the cardinality of our languages learning becomes possible. [Grieser,Jantke,Lange,Thomas:00]. Regular Expressions, Grammar Induction, FSM, Neural Nets, HMM, Naive Bayes, ILP Hidir Aras, Digitale Medien 15

16 Machine Learning for Adaptive IE (1) Wrapper Induction (supervised): delimiter-based extraction rules derived set of samples (training) learned extraction knowledge structures are formally equivalent to regular grammars or finite state automata (Kusmerick, Muslea et al,1999) do not rely on linguistic constraints, but rather formatting features that implicitly delineate the structure of pieces of data found Hidir Aras, Digitale Medien 16

17 Machine Learning for Adaptive IE (2) Automatic Data Extraction (unsupervised): Patricia Trees + rule induction (Chang et al, 2001) Grammar inference (generalization based on ACME), based on tag structure analysis of sample pages of a given class, i.e. bookstore data (Crescenzi et al,2001) Hidir Aras, Digitale Medien 17

18 ACME (1) ACME(Align, Collapse under Mismatch and Extract) The matching algorithm works on two objects (as XHTML) at a time: the sample and a wrapper, i.e. a union-free regular expression one of the given 2 HTML pages is used as the initial version of the wrapper Wrapper is progressively refined trying to find a common regular expression for the two pages this is done by solving mismatches between wrapper and the sample Hidir Aras, Digitale Medien 18

19 ACME (2) A mismatch happens when some token in the sample does not comply to the grammar specified by the wrapper The mismatch is solved by trying to generalize the wrapper The algorithm succeeds if a common wrapper can be generated by solving all mismatches encountered during the parsing. Hidir Aras, Digitale Medien 19

20 ACME (3) : Example Wrapper (initially Page 1) 01: <HTML> 02: Books of: 03: 04: John Smith 05: 06: <UL> 07: <LI> 08-10: Title: 11: DB Primer 12: </LI> 13: <LI> 14-16: Title: 17: Comp. Sys. 18: </LI> 19: </UL> 20: </HTML> Sample (Page 2) Parsing 01: <HTML> 02: Books of: string mismatch (#PCDATA) 03: 04: Paul Jones 05: tag mismatch (?) 06: <IMG src=.../> 07: <UL> 08: <LI> string mismatch (#PCDATA) 09-11: Title: 12: XML at Work 13: </LI> 14: <LI> string mismatch (#PCDATA) 15-17: Title: 18: HTML Scripts tag mismatch (+) 19: </LI> 20: <LI> terminal tag search and 21-23: Title: square matching 24: JavaScript 25: </LI> 26: </UL> square matching Hidir Aras, Digitale Medien 27: </HTML> (backwards) 20

21 ACME (4): The generalized Wrapper Wrapper after solving mismatches: <HTML>Books of:#pcdata ( <IMG src=.../> )? <UL> ( <LI>Title:#PCDATA</LI> )+ </UL> </HTML> Hidir Aras, Digitale Medien 21

22 Example: Using PAT Trees to encode HTML HTML sample code: Congo242 Egypt20 $ Encoded binary string: Code table: 000 001 010 011 100 TEXT Prefix-Search - Regex Search - Range Search - Maximal Repeats Hidir Aras, Digitale Medien 22

23 Semi-infinte Strings (sistrings) Hidir Aras, Digitale Medien 23

24 Multiple string alignment use tag classification (for filtering, abtraction etc.) block-level tags (H-H6,P, DIV, TABLE etc text-level tags (CITE,STRONG, A, IMG, FONT etc) (multiple) string alignment (can ve solved via Dynamic Programming) a d c w b d a d c x b - a d c x b d a d c [w x] b [d -] Hidir Aras, Digitale Medien 24

25 What can be wrapped how? semi-structures data (HTML) rule-based data extraction (manual or semiautomatic) wrapper induction (supervised) tree-based pattern-matching (unsupervised) unstructured text ontology-based grammars (rule-based) standard NLP methods (supervised) Hidir Aras, Digitale Medien 25

26 A qualitative analysis (Laender et al, 2002) Degree of Flexibility Resilience / Adaptiveness Ontologybased Text NLP-based Modelingbased Non- HTML Languages for Wrapper Development Wrapper Induction HTML HTML-aware Degree of Automation Manual Semi-automatic Automatic Hidir Aras, Digitale Medien 26