Web Mining by means of Concept Lattice Theory. Dr. Joyee Yi Zhao FernUniversität in Hagen

Transcription

1 Web Mining by means of Concept Lattice Theory Dr. Joyee Yi Zhao FernUniversität in Hagen 1

2 Outline * Data mining Web mining Concept Lattice Theory Concept Lattices based web mining Conclusion 2

3 Why Is Data Mining Hot? Data mining (knowledge discovery in databases) Extraction of interesting (non-trivial, implicit, previously unknown and potentially useful) information (knowledge) or patterns from data in large databases or other information repositories Necessity is the mother of invention Data is everywhere data mining should be everywhere, too! Understand and use data an imminent task! 3

4 Data Mining: Confluence of Multiple Disciplines Database Technology Statistics Machine Learning (AI) Data Mining Visualization Information Science Other Disciplines 4

5 5

6 Principle of KDD How can the data help solve my problem? What's hiding in there? Define Problem A B I E F D G C H Wow! I did not know that. Mine Data Apply to Problem 6

7 Data Mining Techniques Association Rules Sequential Patterns Classification Clustering Similar Images Outlier Discovery Text/Web Mining 7

8 Recent Progress of R&D in Data Mining Multi-dimensional data analysis: Data warehouse and OLAP (on-line analytical processing) Association, correlation, and causality analysis Sequential patterns and time-series analysis Classification: scalability, associative classification, etc. Clustering and outlier analysis Similarity analysis: curves, trends, images, texts, etc. Text mining, Web mining and Weblog analysis Spatial, multimedia, scientific data mining Data preprocessing and database compression Visual data mining, invisible data mining, etc. 8

9 Association Rules Given: A database of customer transactions Each transaction is a set of items Find all rules X => Y that correlate the presence of one set of items X with another set of items Y Example: 98% of people who purchase diapers and baby food also buy beer. Any number of items in the consequent/antecedent of a rule Possible to specify constraints on rules (e.g., find only rules involving expensive imported products) 9

10 Association Rules(c.) Sample Applications Market basket analysis Attached mailing in direct marketing Fraud detection for medical insurance Department store floor/shelf planning 10

11 Sequential Patterns Given: A sequence of customer transactions Each transaction is a set of items Find all maximal sequential patterns supported by more than a user-specified percentage of customers Example: 10% of customers who bought a PC did a memory upgrade in a subsequent transaction 10% is the support of the pattern Apriori style algorithm can be used to compute frequent sequences 11

12 Classification Given: Database of tuples, each assigned a class label Develop a model/profile for each class Example profile (good credit): (25 <= age <= 40 and income > 40k) or (married = YES) Sample applications: Credit card approval (good, bad) Bank locations (good, fair, poor) Treatment effectiveness (good, fair, poor) 12

13 Clustering Given: Data points and number of desired clusters K Group the data points into K clusters Data points within clusters are more similar than across clusters Sample applications: Customer segmentation Market basket customer analysis Attached mailing in direct marketing Clustering companies with similar growth 13

14 Similar Images Given: A set of images Find: All images similar to a given image All pairs of similar images Sample applications: Medical diagnosis Weather predication Web search engine for images E-commerce 14

15 Outlier Discovery Given: Data points and number of outliers (= n) to find Find top n outlier points Outliers are considerably dissimilar from the remainder of the data Sample applications: Credit card fraud detection Telecom fraud detection Customer segmentation Medical analysis 15

16 Outline Data mining * Web mining Concept Lattice Theory Concept Lattices based web mining Conclusion 16

17 Web Mining: Challenges Today s search engines are plagued by problems: the abundance problem (99% of data of no interest to 99% of people) limited coverage of the Web (Internet sources hidden behind search interfaces) limited query interface based on keyword-oriented search limited customization to individual users 17

18 Web is.. The web is a huge collection of documents Semistructured (ambiguous structure,html, XML) Hyper-link information Access and usage information Dynamic (i.e. New pages are constantly being generated) 18

19 Web Mining 19

20 Web Content Mining Web Mining Extract concept hierarchies/relations from the web Automatic categorization Describe the automatic search of information resources available on-line Web Usage Mining Trend analysis (i.e web dynamics info) Web access association/sequential pattern analysis Data from server access logs, user registration or profiles, user sessions or transactions etc. Web Structure Mining Mine the web document s structures and links 20

21 Semantic Web Tim Berners-Lee, inventor of WWW, URI, HTTP and HTML Next generation of the current web Enrich the web by machine processable information which is organized on different levels XML RDF Ontologies Topic Maps 21

22 XML(eXtensible Markup Language) XML & HTML XML supports the electronic exchange of machine readable documents HTML is designed primarily for human-readable documents XML data shares many features of semistructured data its structure is irregular, and is not always known ahead of time, and can change frequently and without notice easy to convert data from any source into XML 22

23 RDF(Resource Description Framework) An XML-based language for describing information contained in a web resource Triple Subject A Property C Object B RDF schema -- a simple datatyping model for RDF 23

24 Ontologies Meta-data schemas Providing a controlled vocabulary of concepts, each with an explicitly defined and machine processable semantics A successful approach for structuring informal, semi-formal and formal knowledge 24

25 Topic Maps Designed to solve the problem of large quantities of unorganized information Online equivalent of printed indexes Allow users to create a large quantity of metadata and tightly interconnected data 25

26 Semantic Web Mining = Semantic Web + Web Mining Improve the results of web mining by exploiting the new semantic structure in the web Exploit web mining for building the semantic web 26

27 Outline Data mining Web mining * Concept Lattice Theory Concept Lattices based web mining Conclusion 27

28 Concept Lattice Theory Group objects into classes that materialise concepts of the domain under study a set of objects E the relative properties E a binary relation R partial order on concepts let ' ' C1 < C2 X 1 X 2 X 2 X1 Hasse diagram: generalization/specification relationship ' R E C 1 = ( X1, X 1' ), C2 = ( X 2, X '2) E' 28

29 Illustration of a Concept Lattice R a b c A matrix data mining context and its Hasse diagram(concept lattice) 29

30 Why can Concept Lattices support knowledge discovery in databases? Knowledge discovery information discovery combined with knowledge creation representation of information to make the inherent logical structure of the information transparent Concept the logical structure of information is based on concepts and concept systems Concept Lattice as mathematical abstraction of concept system can support humans to discover information and then to create knowledge 30

31 Outline Data mining Web mining Concept Lattice Theory * Concept Lattices based web mining Conclusion 31

32 Concept Lattice Theory based Web Mining Research Structure of Concept Lattices Web mining by means of Concept Lattices normal web mining semantic web mining 32

33 Simplify structure of concept lattices - Pruned concept lattice - Hierarchical concept lattice - Tree 33

34 Web Mining Research based on Concept Lattices Agent design to improve the performance of search engine User browsing behavior extraction and prediction Analyze the structural content of web pages through exploiting the latent information given by HTML tags Extraction of semantic information from unstructured and semi-structured text Combination of normal data mining techniques and Concept Lattice techniques for web mining 34

35 Semantic Web Mining Research based on Concept Lattices A Concept Lattice itself is a kind of semantics Interesting research: Using CL based (clustering) techniques to generate class hierarchies expressible as RDF schema, ontologies Using semantic structure by means of CL into web content, structure and usage mining 35

36 Outline Data mining Web mining Concept lattice Concept Lattice based web mining * Conclusion 36

37 Thank you!!! 37

38 Useful links