Machine Learning for NLP

Transcription

1 Natural Language Processing SoSe 2015 Machine Learning for NLP Dr. Mariana Neves May 4th, 2015 (based on the slides of Dr. Saeedeh Momtazi)

2 Introduction Field of study that gives computers the ability to learn without being explicitly programmed Arthur Samuel, 1959 ( 2

3 Introduction Learning Methods Supervised learning 3 Active learning Unsupervised learning Semi-supervised learning Reinforcement learning

4 Outline 4 Supervised Learning Semi-supervised learning Unsupervised learning

5 Supervised Learning Example: mortgage credit decision Age Income 5

6 Supervised Learning age? income 6

7 Classification Training T1 T2 Tn C1 C2 Cn F1 F2 Fn Model(F,C) Testing Tn+1 7? Fn+1 Cn+1

8 Applications Problems POS tagging Named entity recognition Word sense disambiguation Spam mail detection Language identification Text categorization Information retrieval 8 Items Word Word Word Document Document Document Document Categories POS Named entity Word's sense Spam/Not Spam Language Topic Relevant/Not relevant

9 Part-of-speech tagging 9

10 Named entity recognition 10

11 Word sense disambiguation 11

12 Spam mail detection 12

13 Language identification 13

14 Text categorization 14

15 Classification Training T1 T2 Tn C1 C2 Cn F1 F2 Fn Model(F,C) Testing Tn+1 15? Fn+1 Cn+1

16 Classification algorithms 16 K Nearest Neighbor Support Vector Machines Naïve Bayes Maximum Entropy Linear Regression Logistic Regression Neural Networks Decision Trees Boosting...

17 Classification algorithms 17 K Nearest Neighbor Support Vector Machines Naïve Bayes Maximum Entropy Linear Regression Logistic Regression Neural Networks Decision Trees Boosting...

18 K Nearest Neighbor? 18

19 K Nearest Neighbor? 19

20 K Nearest Neighbor 20 1-nearest neighbor

21 K Nearest Neighbor 3-nearest neighbors? 21

22 K Nearest Neighbor 22 3-nearest neighbors

23 K Nearest Neighbor Simple approach No black-box 23 Choose Features Distance metrics Value of k (majority voting)?

24 K Nearest Neighbor Class distribution is skewed? 24

25 Classification algorithms 25 K Nearest Neighbor Support Vector Machines Naïve Bayes Maximum Entropy Linear Regression Logistic Regression Neural Networks Decision Trees Boosting...

26 Support vector machines 26

27 Support vector machines 27 Find a hyperplane in the vector space that separates the items of the two categories

28 Support vector machines 28 There might be more than one possible separating hyperplane

29 Support vector machines Find the hyperplane with maximum margin Vectors at the margins are called support vectors ( 29

30 Support vector machines Usually provides good results There are libraries available for many programming languages Linear and non-linear classification ( 30

31 Support vector machines Black-box Multi-class problems usually modelled as many binary classifications ( 31

32 Classification algorithms 32 K Nearest Neighbor Support Vector Machines Naïve Bayes Maximum Entropy Linear Regression Logistic Regression Neural Networks Decision Trees Boosting...

33 Naïve Bayes Selecting the class with highest probability Minimizing the number of items with wrong labels c =argmax c P (c i ) i Probability should depend on the to be classified data (d) P(c i d ) 33

34 Naïve Bayes c =argmax c P (c i ) i c =argmax c P (c i d ) i P(d c i ) P(c i ) c =argmax c P (d ) i c =argmax c P (d c i ) P (c i ) i 34

35 Naïve Bayes c =argmax c P (d c i ) P (c i ) i Prior probability Likelihood probability 35

36 Classification Training T1 T2 Tn C1 C2 Cn F1 F2 Fn Model(F,C) Testing Tn+1 36? Fn+1 Cn+1

37 Spam mail detection Features: - words - sender's - contains links - contains attachments - contains money amounts... 37

38 Feature selection Bag-of-words: Document represented by the set of words High dimensional feature space Computationally expensive ( 38

39 Feature selection Solution: Feature selection to select informative words ( 39

40 Feature selection methods 40 Information gain Mutual information χ-square

41 Information gain 41 Measuring the presence or absence of a term in the document Removing words whose information gain is less than a predefined threshold (

42 Information gain IG (w)= i=1 K P (ci ) log P (c i ) + P ( w) i=1 +P(w ) i=1 42 K P (ci w ) log P (ci w) K P (ci w ) log P (ci w )

43 Information gain N = # docs N i = # docs in category ci N w = # docs containing w N w = # docs not containing w N iw = # docs in category ci containing w N i w = # docs in category ci not containing w Ni P(c i )= N 43 Nw P( w)= N P(c i w)= N iw Ni N w P( w )= N P (c i w )= N i w Ni

44 Mutual information Measuring the effect of each word in predicting the category P (w, c i ) MI ( w, c i )=log P (w) P (c i ) 44

45 Mutual information Removing words whose mutual information is less than a predefined threshold MI ( w)=max i MI ( w, c i ) MI ( w)= i P (c i ) MI ( w, c i ) 45

46 χ-square Measuring the dependencies between words and categories 2 N ( N iw N iw N i w N i w ) χ 2 (w, c i )= ( N iw + N i w ) ( N i w + N iw ) ( N iw + N i w ) ( N i w + N iw ) Ranking words based on their χ-square measure χ 2 (w)= i=1 46 K P (c i ) χ 2 (w, ci ) Selecting the top words as features

47 Feature selection 47 These models perform well for document-level classification Spam Mail Detection Language Identification Text Categorization Word-level Classification might need another types of features Part-of-speech tagging Named Entity Recognition

48 Supervised learning 48 Shortcoming Relies heavily on annotated data Time consuming and expensive task

49 Supervised learning Active learning Using a minimum amount of annotated data Annotating further data by human, if they are very informative 49

50 Active learning 50

51 Active learning - Annotating a small amount of data 51

52 Active learning H L M 52 - Calculating the confidence score of the classifier on unlabeled data L

53 Active learning H L M 53 - Finding the informative unlabeled data (data with lowest confidence) L - manually annotating the informative data

54 Outline 54 Supervised Learning Semi-supervised learning Unsupervised learning

55 Semi-supervised learning 55 Annotating data is a time consuming and expensive task Solution Using a minimum amount of annotated data Annotating further data automatically

56 Semi-supervised learning - A small amount of labeled data 56

57 Semi-supervised learning - A large amount of unlabeled data 57

58 Semi-supervised learning - Similarity between the labeled and unlabeled data - Predicting the labels of the unlabeled data 58

59 Semi-supervised learning - Training the classifier using labeled data and predicted labels of unlabeled data 59

60 Semi-supervised learning - Introduce noisy data to the system - Add only predicted label which has high confidence 60

61 Outline 61 Supervised Learning Semi-supervised learning Unsupervised learning

62 Supervised Learning age? income 62

63 Unsupervised Learning age income 63

64 Unsupervised Learning age income 64

65 Clustering 65 Calculating similarities between the data items Grouping similar data items to the same cluster

66 Applications 66 Word clustering Speech recognition Machine translation Named entity recognition Information retrieval... Document clustering Text classification Information retrieval...

67 Speech recognition Computers can recognize a speeech. Computers can wreck a nice peach. recognition speech named-entity hand-writing 67 wreck ball ship

68 Machine translation The cat eats... Die Katze frisst... Die Katze isst... Katze fressen Hund laufen 68 essen Mann Jung

69 Clustering algorithms Flat 69 K-means Hierarchical Top-Down (Divisive) Bottom-Up (Agglomerative) Single-link Complete-link Average-link

70 K-means The best known clustering algorithm (default/baseline), works well for many cases 1 μ = x c x c The cluster center is the mean or centroid of the items in the cluster Minimizing the average squared Euclidean distance of the items from their cluster centers ( 70

71 K-means Initialization: Randomly choose k items as initial centroids while stopping criterion has not been met do for each item do Find the nearest centroid Assign the item to the cluster associated with the nearest centroid end for for each cluster do Update the centroid of the cluster based on the average of all items in the cluster end for end while 71

72 K-means Iterating two steps: Re-assignment Assigning each vector to its closest centroid Re-computation Computing each centroid as the average of the vectors that were assigned to it in re-assignment ( 72

73 K-means 73

74 Hierarchical Agglomerative Clustering (HAC) Creating a hierarchy in the form of a binary tree 74

75 Hierarchical Agglomerative Clustering (HAC) 75 Creating a hierarchy in the form of a binary tree

76 Hierarchical Agglomerative Clustering (HAC) Initial Mapping: Put a single item in each cluster while reaching the predefined number of clusters do for each pair of clusters do Measure the similarity of two clusters end for Merge the two clusters that are most similar end while 76

77 Hierarchical Agglomerative Clustering (HAC) 77 Measuring the similarity in three ways: Single-link Complete-link Average-link

78 Hierarchical Agglomerative Clustering (HAC) Single-link / single-linkage clustering 78 Based on the similarity of the most similar members

79 Hierarchical Agglomerative Clustering (HAC) Complete-link / complete-linkage clustering 79 Based on the similarity of the most dissimilar members

80 Hierarchical Agglomerative Clustering (HAC) Average-link / average-linkage clustering 80 Based on the average of all similarities between the members

81 Hierarchical Agglomerative Clustering (HAC) 81

82 Further reading 82

83 Further reading 83

84 Further reading 84