Machine Learning: Theory and Applications

Transcription

1 Machine Learning: Theory and Applications Zhi-Qiang LIU City University of Hong Kong, Hong Kong, SAR, CHINA Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

2 Topics to be covered (syllabus) Overview and Definitions, Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

3 Topics to be covered (syllabus) Overview and Definitions, Data representations and Modeling, Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

4 Topics to be covered (syllabus) Overview and Definitions, Data representations and Modeling, Dimensionality, Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

5 Topics to be covered (syllabus) Overview and Definitions, Data representations and Modeling, Dimensionality, Learning paradigms, Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

6 Topics to be covered (syllabus) Overview and Definitions, Data representations and Modeling, Dimensionality, Learning paradigms, Symbolic learning methods, Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

7 Topics to be covered (syllabus) Overview and Definitions, Data representations and Modeling, Dimensionality, Learning paradigms, Symbolic learning methods, Neural-Fuzzy learning systems, Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

8 Topics to be covered (syllabus) Overview and Definitions, Data representations and Modeling, Dimensionality, Learning paradigms, Symbolic learning methods, Neural-Fuzzy learning systems, Kernel-based learning, Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

9 Topics to be covered (syllabus) Overview and Definitions, Data representations and Modeling, Dimensionality, Learning paradigms, Symbolic learning methods, Neural-Fuzzy learning systems, Kernel-based learning, Research topics, Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

10 Topics to be covered (syllabus) Overview and Definitions, Data representations and Modeling, Dimensionality, Learning paradigms, Symbolic learning methods, Neural-Fuzzy learning systems, Kernel-based learning, Research topics, Applications. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

11 References Vapnik, V. Statistical Learning Theory, John Wiley, 1998, NY Pearl, J. Probabilistic Reasoning in Intelligent Systems: networks of plausible inference, Morgan Kaufmann, Mitchell T.M. Machine Learning, McGraw-Hill, Gaines, B. and Boose, J. Machine Learning and Uncertainty Reasoning, Academic Press, Bishop, C. Neural Networks for Pattern Recognition, Oxford University Press, London, UK Theodoridis, S. and Koutroumbas, K. Pattern Recognition, Academic Press, Prentice-Hall, Kosko, B. Fuzzy Thinking, Harper Collins, Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

12 Relevant Journals IEEE Transactions on Systems, Man, and Cybernetics, IEEE Transactions on Fuzzy Systems, IEEE Transactions on Neural Networks, IEEE Transactions on Evolutionary Computation, IEEE Transactions on Pattern Analysis and Machine Intelligence, Machine Learning, Artificial Intelligence, Pattern Recognition International Journal of Pattern Recognition and Artificial Intelligence, Neural Networks, and many more. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

13 What is Machine Learning? Learning defined by the Oxford dictionary: To get knowledge of (a subject) or skill (an art, etc) by study, experience or teaching. Also to commit to memory... Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

14 What is Machine Learning? Learning is an essential ability and feature of an intelligent system. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

15 What is Machine Learning? Learning is an essential ability and feature of an intelligent system. It is save to say that learning ability is vital to the survival of species. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

16 What is Machine Learning? Learning is an essential ability and feature of an intelligent system. It is save to say that learning ability is vital to the survival of species. Any autonomous system must be able to learn and adapt. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

17 What is Machine Learning? Learning is an essential ability and feature of an intelligent system. It is save to say that learning ability is vital to the survival of species. Any autonomous system must be able to learn and adapt. In machine learning, it is important to relate to the human learning mechanism. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

18 What is Machine Learning? Learning is an essential ability and feature of an intelligent system. It is save to say that learning ability is vital to the survival of species. Any autonomous system must be able to learn and adapt. In machine learning, it is important to relate to the human learning mechanism. Note: We are not interested in store-and-retrieve type of learning. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

19 What is Machine Learning? Learning is an essential ability and feature of an intelligent system. It is save to say that learning ability is vital to the survival of species. Any autonomous system must be able to learn and adapt. In machine learning, it is important to relate to the human learning mechanism. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

20 Task Performance-related Definition Loosely: To learn is to change and to change is to learn. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

21 Task Performance-related Definition Loosely: To learn is to change and to change is to learn. Learning is any change in a system that allows it to perform better the second time on repetition of the same task or another task drawn from the same population. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

22 Knowledge-gain Learning Definition Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

23 Knowledge-gain Learning Definition Learning can be defined operationally to mean the ability to acquire new skills that could perform new tasks or perform old tasks better (faster, more accurately, etc.) as a result of knowledge acquired by the learning process. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

24 Two Aspects To summarise machine learning can be viewed as having two aspects: The acquisition of new knowledge from external sources; Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

25 Two Aspects To summarise machine learning can be viewed as having two aspects: The acquisition of new knowledge from external sources; The improvement of knowledge representation and structure so that existing knowledge may be better exploited. Lecture slides for Machine Learning, c Zhi-Qiang Liu, 2005 p.

26 State of the Art still modest achievements in ML

27 State of the Art still modest achievements in ML mostly isolated solutions to date

28 State of the Art still modest achievements in ML mostly isolated solutions to date but can now:

29 State of the Art still modest achievements in ML mostly isolated solutions to date but can now: assist automatic knowledge acquisition

30 State of the Art still modest achievements in ML mostly isolated solutions to date but can now: assist automatic knowledge acquisition extract relevant knowledge from large knowledge bases

31 State of the Art still modest achievements in ML mostly isolated solutions to date but can now: assist automatic knowledge acquisition extract relevant knowledge from large knowledge bases abstract higher-level concepts out of data sets

32 State of the Art still modest achievements in ML mostly isolated solutions to date but can now: assist automatic knowledge acquisition extract relevant knowledge from large knowledge bases abstract higher-level concepts out of data sets... etc.

33 State of the Art The recent trend is to build integrated systems:

34 State of the Art The recent trend is to build integrated systems: combine various learning methods

35 State of the Art The recent trend is to build integrated systems: combine various learning methods induction, deduction, analogy, and abduction

36 State of the Art The recent trend is to build integrated systems: combine various learning methods induction, deduction, analogy, and abduction queries, apprenticeship, and abstraction

37 State of the Art The recent trend is to build integrated systems: combine various learning methods induction, deduction, analogy, and abduction queries, apprenticeship, and abstraction symbolic ML, neural networks, generic algorithms: soft computing.

38 State of the Art The recent trend is to build integrated systems: combine various learning methods induction, deduction, analogy, and abduction queries, apprenticeship, and abstraction symbolic ML, neural networks, generic algorithms: soft computing.

39 Outline Learning processes; Learning methods; Learning paradigms; Knowledge representation.

40 Learning as Search Search through a hypothesis space - the space of potential concept descriptions;

41 Learning as Search Search through a hypothesis space - the space of potential concept descriptions; Language to describe the large (possibly infinite) set of concepts;

42 Learning as Search Search through a hypothesis space - the space of potential concept descriptions; Language to describe the large (possibly infinite) set of concepts; Learning algorithm must search in this space in an efficient manner;

43 Learning as Search Search through a hypothesis space - the space of potential concept descriptions; Language to describe the large (possibly infinite) set of concepts; Learning algorithm must search in this space in an efficient manner; Difficulty: How to ignore vast majority of invalid descriptions?

44 Learning as Search Search through a hypothesis space - the space of potential concept descriptions; Language to describe the large (possibly infinite) set of concepts; Learning algorithm must search in this space in an efficient manner; Difficulty: How to ignore vast majority of invalid descriptions? Heuristic methods needed to prune the search.

45 Symbolic vs Numeric ML is traditionally symbolic;

46 Symbolic vs Numeric ML is traditionally symbolic; Symbolic: e.g., [color = orange] rather than [wavelength = 600nm];

47 Symbolic vs Numeric ML is traditionally symbolic; Symbolic: e.g., [color = orange] rather than [wavelength = 600nm]; Numeric: e.g., [object = wing] cf [area = 1.2][angle = 45][...];

48 Symbolic vs Numeric ML is traditionally symbolic; Symbolic: e.g., [color = orange] rather than [wavelength = 600nm]; Numeric: e.g., [object = wing] cf [area = 1.2][angle = 45][...]; Concepts are inherently symbolic;

49 Symbolic vs Numeric ML is traditionally symbolic; Symbolic: e.g., [color = orange] rather than [wavelength = 600nm]; Numeric: e.g., [object = wing] cf [area = 1.2][angle = 45][...]; Concepts are inherently symbolic; An open problem: symbolic or numeric representation?

50 Knowledge Representation Numerous representation schemes:

51 Knowledge Representation Numerous representation schemes: decision trees

52 Knowledge Representation Numerous representation schemes: decision trees propositional calculus

53 Knowledge Representation Numerous representation schemes: decision trees propositional calculus neural networks

54 Knowledge Representation Numerous representation schemes: decision trees propositional calculus neural networks classic logic

55 Knowledge Representation Numerous representation schemes: decision trees propositional calculus neural networks classic logic rules

56 Knowledge Representation Numerous representation schemes: decision trees propositional calculus neural networks classic logic rules frames

57 Knowledge Representation Numerous representation schemes: decision trees propositional calculus neural networks classic logic rules frames associative (semantic) networks

58 Knowledge Representation Numerous representation schemes: decision trees propositional calculus neural networks classic logic rules frames associative (semantic) networks objects

59 Knowledge Representation Numerous representation schemes: decision trees propositional calculus neural networks classic logic rules frames associative (semantic) networks objects cases, examples

60 Knowledge Representation Numerous representation schemes: decision trees propositional calculus neural networks classic logic rules frames associative (semantic) networks objects cases, examples statistics, probabilities

61 Knowledge Representation Numerous representation schemes: decision trees propositional calculus neural networks classic logic rules frames associative (semantic) networks objects cases, examples statistics, probabilities fuzzy logic

62 Knowledge representation most early systems used attribute-value pairs (features): e.g., [color=red or green], [size 2]

63 Knowledge representation most early systems used attribute-value pairs (features): e.g., [color=red or green], [size 2] more recent systems use first-order Horn clauses logic which can include relationships within and between objects: e.g., block(a), block(b), on-top-of(a,b)

64 Taxonomy of ML Symbolic Empirical Learning (SEL)

65 Taxonomy of ML Symbolic Empirical Learning (SEL) supervised (learning from examples)

66 Taxonomy of ML Symbolic Empirical Learning (SEL) supervised (learning from examples) version space

67 Taxonomy of ML Symbolic Empirical Learning (SEL) supervised (learning from examples) version space decision trees (ID3)

68 Taxonomy of ML Symbolic Empirical Learning (SEL) supervised (learning from examples) version space decision trees (ID3) star methodology (AQ11)

69 Taxonomy of ML Symbolic Empirical Learning (SEL) supervised (learning from examples) version space decision trees (ID3) star methodology (AQ11) least generalisation (DLG)

70 Taxonomy of ML Symbolic Empirical Learning (SEL) supervised (learning from examples) version space decision trees (ID3) star methodology (AQ11) least generalisation (DLG) inductive logic programming (FOIL)

71 Taxonomy of ML Symbolic Empirical Learning (SEL) supervised (learning from examples) version space decision trees (ID3) star methodology (AQ11) least generalisation (DLG) inductive logic programming (FOIL) unsupervised (observation and discovery)

72 Taxonomy of ML Symbolic Empirical Learning (SEL) supervised (learning from examples) version space decision trees (ID3) star methodology (AQ11) least generalisation (DLG) inductive logic programming (FOIL) unsupervised (observation and discovery) conceptual clustering (Cobweb)

73 Taxonomy of ML Symbolic Empirical Learning (SEL) supervised (learning from examples) version space decision trees (ID3) star methodology (AQ11) least generalisation (DLG) inductive logic programming (FOIL) unsupervised (observation and discovery) conceptual clustering (Cobweb) discovery (Bacon)

74 Taxonomy of ML Analytical/explanation-based learning

75 Taxonomy of ML Analytical/explanation-based learning learning composite rules (EGGS)

76 Taxonomy of ML Analytical/explanation-based learning learning composite rules (EGGS) learning search control knowledge (LEX)

77 Taxonomy of ML Analytical/explanation-based learning learning composite rules (EGGS) learning search control knowledge (LEX) Analogy, case-based reasoning and exemplars

78 Taxonomy of ML Analytical/explanation-based learning learning composite rules (EGGS) learning search control knowledge (LEX) Analogy, case-based reasoning and exemplars case-based reasoning (CHEF)

79 Taxonomy of ML Analytical/explanation-based learning learning composite rules (EGGS) learning search control knowledge (LEX) Analogy, case-based reasoning and exemplars case-based reasoning (CHEF) exemplars (PROTOS)

80 Taxonomy of ML integrated learning systems

81 Taxonomy of ML integrated learning systems combine various learning techniques (IOU)

82 Taxonomy of ML integrated learning systems combine various learning techniques (IOU) learning apprentice systems (LEAP)

83 Taxonomy of ML integrated learning systems combine various learning techniques (IOU) learning apprentice systems (LEAP) Sub-symbolic learning systems

84 Taxonomy of ML integrated learning systems combine various learning techniques (IOU) learning apprentice systems (LEAP) Sub-symbolic learning systems generic algorithms

85 Taxonomy of ML integrated learning systems combine various learning techniques (IOU) learning apprentice systems (LEAP) Sub-symbolic learning systems generic algorithms neural networks

86 Taxonomy of ML integrated learning systems combine various learning techniques (IOU) learning apprentice systems (LEAP) Sub-symbolic learning systems generic algorithms neural networks fuzzy adaptive systems

87 Taxonomy of ML integrated learning systems combine various learning techniques (IOU) learning apprentice systems (LEAP) Sub-symbolic learning systems generic algorithms neural networks fuzzy adaptive systems kernel-based approaches: SVM.

88 Taxonomy of ML integrated learning systems combine various learning techniques (IOU) learning apprentice systems (LEAP) Sub-symbolic learning systems generic algorithms neural networks fuzzy adaptive systems kernel-based approaches: SVM.

89 Taxonomy of ML integrated learning systems combine various learning techniques (IOU) learning apprentice systems (LEAP) Sub-symbolic learning systems generic algorithms neural networks fuzzy adaptive systems kernel-based approaches: SVM.