WEB MINING INTRO.

Transcription

1 1 WEB MINING INTRO

2 Outline 2 Introduction Web Mining Web Content Mining Web Structure Mining Web Usage Mining Conclusion & Exam Questions

3 Four Problems 3 Finding relevant information Low precision and unindexed information Creating new knowledge out of available information on the web Personalizing the information Catering to personal preference in content and presentation Learning about the consumers What does the customer want to do? Using web data to effectively market products and/or services

4 Other Approaches 4 Web mining is NOT the only approach Database approach (DB) Information retrieval (IR) Natural language processing (NLP) In-depth syntactic and semantic analysis Web document community Standards, manually appended meta-information, maintained directories, etc

5 Direct vs. Indirect Web Mining 5 Web mining techniques can be used to solve the information overload problems: Directly Attack the problem with web mining techniques E.g. newsgroup agent classifies news as relevant Indirectly Used as part of a bigger application that addresses problems E.g. used to create index terms for a web search service

6 The Research 6 Converging research from: Database, information retrieval, and artificial intelligence (specifically NLP and machine learning) Focusing on research from the machine learning point of view

7 Outline 7 Introduction Web Mining Web Content Mining Web Structure Mining Web Usage Mining Conclusion & Exam Questions

8 Web Mining: Definition 8 Web mining refers to the overall process of discovering potentially useful and previously unknown information or knowledge from the Web data. Can be viewed as four subtasks Not the same as Information Retrieval Not the same as Information Extraction www Knowledge

9 Web Mining: Subtasks 9 Resource finding Retrieving intended documents Information selection/pre-processing Select and pre-process specific information from selected documents Generalization Discover general patterns within and across web sites Analysis Validation and/or interpretation of mined patterns

10 Web Mining: Not IR 10 Information retrieval (IR) is the automatic retrieval of all relevant documents while at the same time retrieving as few of the non-relevant documents as possible Web document classification, which is a Web Mining task, could be part of an IR system (e.g. indexing for a search engine)

11 Web Mining: Not IE 11 Information extraction (IE) aims to extract the relevant facts from given documents IE systems for the general Web are not feasible Most focus on specific Web sites or content

12 Web Mining and Machine Learning 12 Machine learning is concerned with the development of algorithms and techniques that allow computers to "learn". Web mining is NOT learning from the Web. Some applications of machine learning on the web are NOT Web Mining Methods used for Web Mining are NOT limited to machine learning Oops, there is a close relationship between web mining and machine learning

13 Web Mining: The Agent Paradigm 13 User Interface Agents information retrieval agents, information filtering agents, & personal assistant agents. Distributed Agents distributed agents for knowledge discovery or data mining. Problem solving by a group of agents Mobile Agents

14 Web Mining: The Agent Paradigm 14 Content-based approach The system searches for items that match based on an analysis of the content using the user preferences. Collaborative approach The system tries to find users with similar interests Recommendations given based on what similar users did

15 Outline 15 Introduction Web Mining Web Content Mining Web Structure Mining Web Usage Mining Conclusion & Exam Questions

16 Web Mining Categories 16 Web Content Mining Discovering useful information from web contents/data/documents. Web Structure Mining Discovering the model underlying link structures (topology) on the Web. E.g. discovering authorities and hubs Web Usage Mining Make sense of data generated by surfers Usage data from logs, user profiles, user sessions, cookies, user queries, bookmarks, mouse clicks and scrolls, etc.

17 Web Content Data Structure 17 Unstructured free text Semi-structured HTML More structured Table or Database generated HTML pages Multimedia data receive less attention than text or hypertext

18 18

19 Web Mining Taxonomy Web Mining Web Content Mining Web Structure Mining Web Usage Mining Identify information within given web pages Distinguish personal home pages from other web pages Uses interconnections between web pages to give weight to the pages Understand access patterns and the trends to improve structure

20 Outline 20 Introduction Web Mining Web Content Mining Web Structure Mining Web Usage Mining Conclusion & Exam Questions

21 Web Content Mining: IR View 21 Unstructured Documents Bag of words, or phrase-based feature representation Features can be boolean or frequency based Features can be reduced using different feature selection techniques Word stemming, combining morphological variations into one feature

22 Web Content Mining: IR View 22 Semi-Structured Documents Uses richer representations for features, based on information from the document structure (typically HTML and hyperlinks) Uses common data mining methods (whereas unstructured might use more text mining methods)

23 Web Content Mining: DB View 23 Tries to infer the structure of a Web site or transform a Web site to become a database Better information management Better querying on the Web Can be achieved by: Finding the schema of Web documents Building a Web warehouse Building a Web knowledge base Building a virtual database

24 24

25 Web Content Mining: DB View 25 Mainly uses the Object Exchange Model (OEM) Represents semi-structured data (some structure, no rigid schema) by a labeled graph Process typically starts with manual selection of Web sites for content mining Main application: building a structural summary of semi-structured data (schema extraction or discovery)

26 Web Content Mining Web Content Mining Agent Based Approach Database Approach Intelligent Search Agent Information Filtering & Categorization Personalized Web Agent Multilevel Databases Web Query Systems

27 Outline 27 Introduction Web Mining Web Content Mining Web Structure Mining Web Usage Mining Conclusion & Exam Questions

28 Web Structure Mining 28 Interested in the structure between Web documents (not within a document) Inspired by the study of social networks and citation analysis Example: PageRank Google Application: Discovering micro-communities in the Web Measuring the completeness of a Web site

29 Outline 29 Introduction Web Mining Web Content Mining Web Structure Mining Web Usage Mining Conclusion & Exam Questions

30 Web Usage Mining 30 Tries to predict user behavior from interaction with the Web Wide range of data (logs) Web client data Proxy server data Web server data Two common approaches Map usage data into relational tables before using adapted data mining techniques Use log data directly by utilizing special pre-processing techniques

31 Web Usage Mining 31 Typical problems: Distinguishing among unique users, server sessions, episodes, etc in the presence of caching and proxy servers Often Usage Mining uses some background or domain knowledge E.g. site topology, Web content, etc

32 Web Usage Mining 32 Two main categories: Learning a user profile (personalized) Web users would be interested in techniques that learn their needs and preferences automatically Learning user navigation patterns (impersonalized) Information providers would be interested in techniques that improve the effectiveness of their Web site or biasing the users towards the goals of the site

33 Outline 33 Introduction Web Mining Web Content Mining Web Structure Mining Web Usage Mining Conclusion & Exam Questions

34 Conclusions 34 The paper tried to resolve confusion with regards to the term Web Mining Differentiated from IR and IE Suggest three Web mining categories Content, Structure, and Usage Mining Briefly described approaches for the three categories Explored connection with agent paradigm