Bing Liu. Web Data Mining. Exploring Hyperlinks, Contents, and Usage Data. With 177 Figures. ~ Spring~r

Transcription

1 Bing Liu Web Data Mining Exploring Hyperlinks, Contents, and Usage Data With 177 Figures ~ Spring~r

2 Table of Contents 1. Introduction What is the World Wide Web? ABrief History of the Web and the Internet Web Data Mining What is Data Mining? What is Web Mining? Summary of Chapters How to Read this Book 11 Bibliographie Notes 12 Part I: Data Mining Foundations 2. Association Rules and Sequential Patterns Basic Coneepts of Association Rules Apriori Aigorithm Frequent Itemset Generation Association Rule Generation Data Formats for Assoeiation Rule Mining Mining with Multiple Minimum Supports Extended Model Mining Aigorithm Rule Generation Mining Class Association Rules Problem Definition Mining Aigorithm Mining with Multiple Minimum Supports 37

3 XII Table ofcontents 2.6. Basie Coneepts of Sequential Patterns Mining Sequential Patterns Based on GSp GSP Aigorithm Mining with Multiple Minimum Supports Mining Sequential Patterns Based on PrefixSpan PrefixSpan Aigorithm Mining with Multiple Minimum Supports Generating Rules from Sequential Patterns Sequential Rules Label Sequential Rules Class Sequential Rules 51 Bibliographie Notes Supervised Learning Basie Coneepts Deeision Tree Induetion Learning Aigorithm Impurity Function Handling of Continuous Attributes Some Other Issues Classifier Evaluation Evaluation Methods Precision, Recall, F-score and Breakeven Point Rule Induetion Sequential Covering Rule Learning: Learn-One-Rule Function" Discussion Classifieation Based on Associations Classification Using Class Association Rules Class Association Rules as Features Classification Using Normal Association Rules Na'ive Bayesian Classifieation Na'ive Bayesian Text Classifieation Probabilistic Framework NaIve Bayesian Model Discussion Support Veetor Maehines Linear SVM: Separable Case 99

4 Table ofcontents XIII Linear SVM: Non-Separable Case Nonlinear SVM: Kernel Functions K-Nearest Neighbor Learning Ensemble of Classifiers Bagging Boosting 114 Bibliographie Notes Unsupervised Learning Basie Coneepts K-means Clustering K-means Aigorithm Disk Version of the K-means Aigorithm Strengths and Weaknesses Representation of Clusters Common Ways of Representing Clusters Clusters of Arbitrary Shapes Hierarehieal Clustering Single-Link Method Complete-Link Method Average-Link Method Strengths and Weaknesses Distanee Funetions Numeric Attributes Binary and Nominal Attributes Text Documents Data Standardization Handling of Mixed Attributes Whieh Clustering Aigorithm to Use? Cluster Evaluation Diseovering Holes and Data Regions 146 Bibliographie Notes Partially Supervised Learning Learning from Labeled and Unlabeled Examples EM Algorithm with NaIve Bayesian Classification. 153

5 XIV Table of Contents Co-Training Self-Training Transductive Support Vector Machines Graph-Based Methods Discussion Learning fram Positive and Unlabeled Examples Applications of PU Learning Theoretical Foundation Building Classifiers: Two-Step Approach Building Classifiers: Direct Approach Discussion 178 Appendix: Derivation of EM for Na'ive Bayesian Classification Bibliographie Notes 181 Part 11: Web Mining 6. Information Retrieval and Web Search Basic Concepts of Information Retrieval Information Retrieval Models" Boolean Model Vector Space Model Statistical Language Model Relevanee Feedback Evaluation Measures Text and Web Page Pre-Proeessing Stopword Removal Stemming Other Pre-Processing Tasks for Text Web Page Pre-Processing Duplicate Detection Inverted Index and Its Compression Inverted Index Search Using an Inverted Index Index Construction Index Compression 209

6 Table of Contents XV 6.7. Latent Semantie Indexing SingularValue Deeomposition Query and Retrieval An Example Diseussion '" Web Seareh Meta-Seareh: Combining Multiple Rankings Combination Using Similarity Scores Combination Using Rank Positions Web Spamming Content Spamming Link Spamming Hiding Teehniques Combating Spam 234 Bibliographie Notes Link Analysis Social Network Analysis Centrality Prestige Co-Citation and Bibliographie Coupling Co-Citation Bibliographie Coupling PageRank PageRank Aigorithm Strengths and Weaknesses of PageRank Timed PageRank HITS HITS Aigorithm Finding Other Eigenveetors Relationships with Co-Citation and Bibliographie Coupling Strengths and Weaknesses of HITS Community Diseovery Problem Definition Bipartite Core Communities Maximum Flow Communities Communities Based on Betweenness Overlapping Communities of Named Entities

7 XVI Table ofcontents Bibliographie Notes Web Crawling ABasie Crawler Algorithm Breadth-First Crawlers Preferential Crawlers Implementation Issues Fetehing Parsing Stopword Removal and Stemming Link Extraction and Canonicalization Spider Traps Page Repository Concurrency Universal Crawlers Scalability Coverage vs Freshness vs Importance Foeused Crawlers Topieal Crawlers Topical Locality and Cues Best-First Variations Adaptation Evaluation Crawler Ethies and Confliets Some New Developments Bibliographie Notes Structured Data Extraction: Wrapper Generation' Preliminaries Two Types of Data Rich Pages Data Model HTML Mark-Up Encoding of Data Instances Wrapper Induetion Extraction from a Page Learning Extraction Rules Identifying Informative Examples Wrapper Maintenance 338

8 Table ofcontents XVII 9.3. Instance-Based Wrapper Learning Automatie Wrapper Generation: Problems Two Extraction Problems Patterns as Regular Expressions String Matching and Tree Matching String Edit Distance Tree Matching Multiple Alignment Center Star Method Partial Tree Alignment Building DOM Trees Extraction Based on a Single List Page: Flat Data Records Two Observations about Data Records Mining Data Regions Identifying Data Records in Data Regions Data Item Alignment and Extraction Making Use of Visuallnformation Some Other Techniques Extraction Based on a Single List Page: Nested Data Records Extraction Based on Multiple Pages Using Techniques in Previous Sections RoadRunner Aigorithm Some Other Issues Extraction from Other Pages Disjunction or Optional A Set Type or a Tuple Type Labeling and Integration Domain Specific Extraction Discussion 379 Bibliographie Notes Information Integration Introduction to Schema Matching Pre-Processing tor Schema Matching Schema-Level Match 385

9 XVIII Table of Contents Linguistic Approaches Constraint Based Approaches Domain and Instance-Level Matching Combining Similarities :m Match Some Other Issues Reuse of Previous Match Results Matching a Large Number of Schemas Schema Match Results User Interactions Integration of Web Query Interfaces A Clustering Based Approach A Correlation Based Approach An Instance Based Approach Constructing a Unified Global Query Interface Structural Appropriateness and the Merge Aigorithm Lexical Appropriateness Instance Appropriateness Bibliographie Notes Opinion Mining Sentiment Classification Classification Based on Sentiment Phrases Classification Using Text Classification Methods Classification Using a Score Function Feature-Based Opinion Mining and Summarization Problem Definition Object Feature Extraction Feature Extraction from Pros and Cons of Format Feature Extraction from Reviews of of Formats 2 and Opinion Orientation Classification Comparative Sentence and Relation Mining Problem Definition Identification of Gradable Comparative Sentences 435

10 Table of Contents XIX Extraction of Comparative Relations Opinion Seareh Opinion Spam Objectives and Actions of Opinion Spamming Types of Spam and Spammers Hiding Techniques Spam Detection 444 Bibliographie Notes Web Usage Mining Data Colleetion and Pre-Proeessing Sources and Types of Data Key Elements of Web Usage Data Pre-Processing Data Modeling for Web Usage Mining Diseovery and Analysis of Web Usage Patterns Session and Visitor Analysis Cluster Analysis and Visitor Segmentation Association and Correlation Analysis Analysis of Sequential and Navigational Patterns Classification and Prediction Based on Web User Transactions Diseussion and Outlook 482 Bibliographie Notes, 482 References 485 Index