Distributional Similarity

Transcription

1 Overview and alpage inria & paris vii Séminaire cental Vendredi 5 novembre 2010

2 Outline 1 2 Similarity Context Dimensionality reduction Latent semantic analysis Non-negative matrix factorization Tensors 3

3 Semantic similarity Most work on semantic similarity relies on the distributional hypothesis (Harris 1954) Take a word and its contexts: tasty tnassiorc greasy tnassiorc tnassiorc with butter tnassiorc for breakfast By looking at a word s context, one can infer its meaning

4 Semantic similarity Most work on semantic similarity relies on the distributional hypothesis (Harris 1954) Take a word and its contexts: tasty tnassiorc greasy tnassiorc tnassiorc with butter tnassiorc for breakfast FOOD By looking at a word s context, one can infer its meaning

5 Semantic similarity Most work on semantic similarity relies on the distributional hypothesis (Harris 1954) Take a word and its contexts: tasty tnassiorc greasy tnassiorc tnassiorc with butter tnassiorc for breakfast By looking at a word s context, one can infer its meaning

6 Matrix Similarity Context Dimensionality reduction Tensors Capture co-occurrence frequencies of two entities

7 Matrix Similarity Context Dimensionality reduction Tensors Capture co-occurrence frequencies of two entities rouge délicieux rapide d occasion pomme vin voiture camion

8 Matrix Similarity Context Dimensionality reduction Tensors Capture co-occurrence frequencies of two entities rouge délicieux rapide d occasion pomme vin voiture camion

9 Matrix Similarity Context Dimensionality reduction Tensors Capture co-occurrence frequencies of two entities rouge délicieux rapide d occasion pomme vin voiture camion

10 Matrix Similarity Context Dimensionality reduction Tensors Capture co-occurrence frequencies of two entities rouge délicieux rapide d occasion pomme vin voiture camion

11 Similarity calculation Similarity Context Dimensionality reduction Tensors Cosine P x y = n i=1 x i y i Pn P n i=1 x2 i i=1 y i 2 Examples: cos(pomme, vin) =.96 cos(pomme, voiture) =.42 cos( x, y ) = x y Other possibilities: set-theoretic measures Dice Jaccard probabilistic measures Kullback-Leibler divergence Jensen-Shannon divergence

12 Different kinds of context Similarity Context Dimensionality reduction Tensors Three different word space models based on context: document-based model (nouns documents) window-based model (nouns context words) syntax-based model (nouns dependency relations) Each model with plethora of parameters! document size, window size, type of dependency relations weighting function ± dimensionality reduction

13 Similarity Context Dimensionality reduction Tensors Different kinds of semantic similarity tight, synonym-like similarity: (near-)synonymous or (co-)hyponymous loosely related, topical similarity: more loose relationships, such as association and meronymy

14 Similarity Context Dimensionality reduction Tensors Different kinds of semantic similarity tight, synonym-like similarity: (near-)synonymous or (co-)hyponymous loosely related, topical similarity: more loose relationships, such as association and meronymy Example médecin doctor : docteur doctor, médecin de famille family doctor, chirurgien surgeon, spécialiste specialist, dermatologue dermatologist, gynécologue gynaecologist médecin doctor : malade patient, maladie disease, diagnostic diagnosis, traitement treatment, hôpital hospital, stéthoscope stethoscope

15 Relation context similarity Similarity Context Dimensionality reduction Tensors Different context leads to different kind of similarity Syntax, small window large window, documents The former models induce tight, synonymous similarity The latter models induce topical relatedness

16 Relation context similarity Similarity Context Dimensionality reduction Tensors Different context leads to different kind of similarity Syntax, small window large window, documents The former models induce tight, synonymous similarity The latter models induce topical relatedness Evaluation Syntax-based model scores best when evaluated according to Wordnet similarity measures (cornetto) Large window and document-based do not score well on Wordnet similarity, but do score on Wordnet domain evaluation

17 Similarity Context Dimensionality reduction Tensors Two reasons for performing dimensionality reduction: Intractable computations When number of elements and number of features is too large, similarity computations may become intractable reduction of the number of features makes computation tractable again Generalization capacity the dimensionality reduction is able to describe the data better, or is able to capture intrinsic semantic features dimensionality reduction is able to improve the results (counter data sparseness and noise)

18 Similarity Context Dimensionality reduction Tensors Latent semantic analysis: introduction Application of a mathematical/statistical technique to simulate how humans learn the semantics of words LSA finds latent semantic dimensions according to which words and documents can be identified Goal: counter data sparseness and get rid of noise

19 Similarity Context Dimensionality reduction Tensors Latent semantic analysis: introduction What is latent semantic analysis technically speaking? The application of singular value decomposition to a term-document matrix to improve similarity calculations

20 Singular value decomposition Similarity Context Dimensionality reduction Tensors Find the dimensions that explain most variance by solving number of eigenvector problems Only keep the n most important dimensions (n = )

21 Similarity Context Dimensionality reduction Tensors SVD: three matrices

22 LSA: Example Similarity Context Dimensionality reduction Tensors LSA in (part of) Twente Nieuws Corpus: 10 years of Dutch newspaper texts (AD, NRC, TR, VK, PAR) terms = nouns documents = paragraphs 20,000 terms * 2,000,000 documents matrix reduced to 300 dimensions

23 LSA: criticism Similarity Context Dimensionality reduction Tensors LSA criticized for a number of reasons: dimensionality reduction is best fit in least-square sense, but this is only valid for normally distributed data; language data is not normally distributed Dimensions may contain negative values; it is not clear what negativity on a semantic scale should designate Shortcomings are remedied by subsequent techniques (PLSA, LDA, NMF,... )

24 Similarity Context Dimensionality reduction Tensors Non-negative matrix factorization: technique Given a non-negative matrix V, find non-negative matrix factors W and H such that: Choosing r n, m reduces data V nxm W nxr H rxm (1) Constraint on factorization: all values in three matrices need to be non-negative values ( 0) Constraint brings about a parts-based representation: only additive, no subtractive relations are allowed

25 Similarity Context Dimensionality reduction Tensors Non-negative matrix factorization: technique Different kinds of NMF s that minimize different cost functions: Square of Euclidean distance Kullback-Leibler divergence better suited for language phenomena To find NMF is to minimize D(V WH) with respect to W and H, subject to the constraints W, H 0 This can be done with update rules H aµ H aµ i W V iµ ia (WH) iµ k W ka W ia W ia µ H aµ V iµ (WH) iµ v H av (2) these update rules converge to a local minimum in the minimization of KL divergence

26 NMF: graphical representation Similarity Context Dimensionality reduction Tensors

27 NMF: results Similarity Context Dimensionality reduction Tensors Context vectors (5k nouns 2k co-occurring nouns) extracted from clef corpus nmf is able to capture clear semantic dimensions Examples: bus bus, taxi taxi, trein train, halte stop, reiziger traveler, perron platform, tram tram, station station, chauffeur driver, passagier passenger bouillon broth, slagroom cream, ui onion, eierdooier egg yolk, laurierblad bay leaf, zout salt, deciliter decilitre, boter butter, bleekselderij celery, saus sauce

28 Two-way vs. three-way Similarity Context Dimensionality reduction Tensors all methods use two way co-occurrence frequencies matrix suitable for two-way problems words documents nouns dependency relations not suitable for n-way problems words documents authors verbs subjects direct objects

29 Two-way vs. three-way Similarity Context Dimensionality reduction Tensors all methods use two way co-occurrence frequencies matrix suitable for two-way problems words documents nouns dependency relations not suitable for n-way problems tensor words documents authors verbs subjects direct objects

30 Similarity Context Dimensionality reduction Tensors Non-negative tensor factorization: technique Idea similar to non-negative matrix factorization Calculations are different min xi R D1 0,y i R D2 0,z i R D3 T k 0 i=1 x i y i z i 2 F

31 NTF: graphical representation Similarity Context Dimensionality reduction Tensors

32 Task: automatic extraction of multi-word expressions (mwes) from large corpora Starting point: many mwes are non-compositional, i.e. the meaning of the mwe is not the sum of the meaning of the individual words Intuition: a noun within a mwe cannot easily be replaced by a semantically similar noun Use of semantic clusters to determine whether a mwe-candidate is compositional or not

33 Intuition break the ice break the vase *snow cup *hail dish In the first expression, it is not possible to replace ice with semantically related nouns such as snow or hail; in the second expression, it is possible to replace vase with semantically related words such as cup or dish.

34 Overview of the method verb + prepositional complement instances are extracted from 500M word corpus (focus on mwe with PP) matrix of 5K verb-preposition combinations 10K nouns is created 10K nouns are automatically clustered using distributional similarity measures number of statistical measures is applied to determine unique associations given the cluster in which a noun appears

35 Measures inspired by selectional preferences (Resnik 1993), entropy-based Kullback-Leibler divergence between P(n) and P(n v) S v = n p(n v) p(n v) log p(n) (3)

36 Measures The preference of the verb for the noun [0, 1] A v n = p(n v) log p(n v) p(n) S v (4) Ratio of verb preference for a particular noun, compared to other nouns in the cluster [0, 1] R v n = A v n n ɛc A v n (5)

37 Measures The preference of the noun for the verb [0, 1] A n v = p(v n) log p(v n) p(v) S n (6) Ratio of noun preference for a particular verb, compared to other nouns in the cluster [0, 1] R n v = A n v n ɛc A n v (7)

38 An elaborated example in de smaak vallen in de put vallen *geur kuil *voorkeur krater *stijl greppel in the taste fall in the well fall to be appreciated to fall down the well *smell hole *preference crater *style trench

39 An elaborated example smaak: idioom, karakter, persoonlijkheid, stijl, temperament, thematiek, uiterlijk, uitstraling, voorkomen mwe candidate (1) (2) (3) (4) mwe? val#in smaak yes val#in karakter no val#in stijl no

40 An elaborated example put: gaatje, gat, kloof, krater, kuil, lek, scheur, valkuil mwe candidate (1) (2) (3) (4) mwe? val#in put no val#in kuil no val#in kloof no

41 Quantitative Evaluation Fully automated, compared to Referentie Bestand Nederlands Upper bound consists of all RBN mwe s present in the data Parameters Prec Rec F-Measure (1) (2) (3) (4) N (%) (%) (%) Fazly/Stevenson Random baseline

42 Conclusion Non-compositionality based algorithm is able to rule out expression that are coined mwe s by traditional algorithms, improving on state-of-the-art Using measures (1) and (2) gives best results; using (3) and (4) increases precision but degrades recall

43 Ambiguity Problem: ambiguity bar

44 Ambiguity Problem: ambiguity bar

45 Ambiguity Problem: ambiguity bar

46 Ambiguity Problem: ambiguity bar

47 Ambiguity Problem: ambiguity bar

48 Ambiguity Problem: ambiguity bar

49 Ambiguity Problem: ambiguity bar Main research question: can topical similarity and tight, synonym-like similarity be combined to differentiate between various senses of a word?

50 Goal: classification of nouns according to both window-based context (with large window) and syntactic context Construct three matrices capturing co-occurrence frequencies for each mode nouns cross-classified by dependency relations nouns cross-classified by (bag of words) context words dependency relations cross-classified by context words Apply nmf to matrices, but interleave the process Result of former factorization is used to initialize factorization of the next one

51 Graphical Representation 5k 80k A nouns x dependency relations k = x W 80k H 5k 2k B nouns x context words k = x V 2k G 2k 80k C context words x dependency relations 2k = x 50 80k U 50 F

52 Sense subtraction switch off dimension(s) of an ambiguous word to reveal other possible senses From matrix W, we know which dimensions are the most important for a certain word Matrix H gives the importance of each dependency relation given a dimension subtract dependency relations that are responsible for a given dimension from the original noun vector v new = v orig ( v 1 h dim ) each dependency relation is multiplied by a scaling factor, according to the load of the feature on the subtracted dimension

53 Combination with clustering A simple clustering algorithm (k-means) assigns ambiguous nouns to its predominant sense Centroid of the cluster is folded into nmf model The dimensions that define the centroid are subtracted from the ambiguous noun vector Adapted noun vector is fed to the clustering algorithm again

54 Experimental Design Approach applied to Dutch, using Twente Nieuws Corpus (± 500M words) Corpus parsed with Dutch dependency parser alpino three matrices constructed with: 5k nouns 80k dependency relations 5k nouns 2k context words 80k dependency relations 2k context words Factorization to 50 dimensions

55 Example dimension: transport 1 nouns: auto car, wagen car, tram tram, motor motorbike, bus bus, metro subway, automobilist driver, trein trein, stuur steering wheel, chauffeur driver 2 context words: auto car, trein train, motor motorbike, bus bus, rij drive, chauffeur driver, fiets bike, reiziger reiziger, passagier passenger, vervoer transport 3 dependency relations: viertraps adj four pedal, verplaats met obj move with, toeter adj honk, tank in houd obj [parsing error], tank subj refuel, tank obj refuel, rij voorbij subj pass by, rij voorbij adj pass by, rij af subj drive off, peperduur adj very expensive

56 Pop: most similar words pop music doll 1 pop, rock, jazz, meubilair furniture, popmuziek pop music, heks witch, speelgoed toy, kast cupboard, servies [tea] service, vraagteken question mark 2 pop, meubilair furniture, speelgoed toy, kast cupboard, servies [tea] service, heks witch, vraagteken question mark sieraad jewel, sculptuur sculpture, schoen shoe 3 pop, rock, jazz, popmuziek pop music, heks witch, danseres dancer, servies [tea] service, kopje cup, house house music, aap monkey

57 Barcelona: most similar words Spanish city Spanish football club 1 Barcelona, Arsenal, Inter, Juventus, Vitesse, Milaan Milan, Madrid, Parijs Paris, Wenen Vienna, München Munich 2 Barcelona, Milaan Milan, München Munich, Wenen Vienna, Madrid, Parijs Paris, Bonn, Praag Prague, Berlijn Berlin, Londen London 3 Barcelona, Arsenal, Inter, Juventus, Vitesse, Parma, Anderlecht, PSV, Feyenoord, Ajax

58 Clustering example: werk 1 werk work, beeld image, foto photo, schilderij painting, tekening drawing, doek canvas, installatie installation, afbeelding picture, sculptuur sculpture, prent picture, illustratie illustration, handschrift manuscript, grafiek print, aquarel aquarelle, maquette scale-model, collage collage, ets etching 2 werk work, boek book, titel title, roman novel, boekje booklet, debuut debut, biografie biography, bundel collection, toneelstuk play, bestseller bestseller, kinderboek child book, autobiografie autobiography, novelle short story, 3 werk work, voorziening service, arbeid labour, opvoeding education, kinderopvang child care, scholing education, huisvesting housing, faciliteit facility, accommodatie acommodation, arbeidsomstandigheid working condition

59 Evaluation: methodology Comparison to EuroWordNet senses using Wu & Palmer s Wordnet similarity measure a sense is assigned to a correct cluster if: for the top 4 words of the cluster the average similarity between the sense and the top 4 words exceeds a similarity threshold in EuroWordNet and the sense has not yet been considered correct when multiple senses exist in EuroWordNet, the one with maximum similarity is chosen

60 Evaluation: precision & recall Precision of a word: Percentage of correct clusters to which it is assigned overall: average precision of all words in test set Recall of a word: Percentage of senses in EuroWordnet that have a corresponding cluster overall: average recall of all words in test set Test set: words covered by algorithm that are also present in EuroWordNet (3683 words)

61 Evaluation: results threshold θ.40 (%).50 (%).60 (%) kmeans nmf prec rec cbc prec rec kmeans orig prec rec

62 Evaluation: results kmeans nmf beats cbc with regard to precision cbc beats kmeans nmf with regard to recall kmeans nmf has higher recall than kmeans orig, so algorithm is able to find multiple senses with good precision

63 Conclusion Combining bag of words data and syntactic data is useful bag of words data (factorized with nmf) puts its finger on topical dimensions syntactic data is particularly good at finding similar words a clustering approach allows one to determine which topical dimension(s) are responsible for a certain sense and adapt the (syntactic) feature vector of the noun accordingly subtracting the more dominant sense to discover less dominant senses Algorithm scores better with regard to precision; lower with regard to recall

64 Standard selectional preference models: two-way co-occurrences Keeping track of single relationships But: two-way selectional preference models are not sufficiently rich Compare: The skyscraper is playing coffee. The turntable is playing the piano.

65 The skyscraper is playing coffee. (play, su, scyscraper) (play, obj, coffee) The turntable is playing the piano. (play, su, turntable) (play, obj, piano) (play, su, turntable, obj, piano)

66 Three-way extraction of selectional preferences Approach applied to Dutch, using twente nieuws corpus (500M words of newspaper texts) parsed with Dutch dependency parser alpino three-way co-occurrence of verbs with subjects and direct objects extracted adapted with extension of pointwise mutual information Resulting tensor 1k verbs 10k subjects 10k direct objects non-negative tensor factorization with k dimensions (k = 50, 100, 300)

67 Graphical representation

68 Example dimension: police action subjects su s verbs v s objects obj s politie police.99 houd aan arrest.64 verdachte suspect.16 agent policeman.07 arresteer arrest.63 man man.16 autoriteit authority.05 pak op run in.41 betoger demonstrator.14 Justitie Justice.05 schiet dood shoot.08 relschopper rioter.13 recherche detective force.04 verdenk suspect.07 raddraaier instigator.13 marechaussee military police.04 tref aan find.06 overvaller raider.13 justitie justice.04 achterhaal overtake.05 Roemeen Romanian.13 arrestatieteam special squad.03 verwijder remove.05 actievoerder campaigner.13 leger army.03 zoek search.04 hooligan hooligan.13 douane customs.02 spoor op track.03 Algerijn Algerian.13

69 Example dimension: legislation subjects su s verbs v s objects obj s meerderheid majority.33 steun support.83 motie motion.63 VVD.28 dien in submit.44 voorstel proposal.53 D66.25 neem aan pass.23 plan plan.28 Kamermeerderheid.25 wijs af reject.17 wetsvoorstel bill.19 fractie party.24 verwerp reject.14 hem him.18 PvdA.23 vind think.08 kabinet cabinet.16 CDA.23 aanvaard accepts.05 minister minister.16 Tweede Kamer.21 behandel treat.05 beleid policy.13 partij party.20 doe do.04 kandidatuur candidature.11 Kamer Chamber.20 keur goed pass.03 amendement amendment.09

70 Example dimension: exhibition subjects su s verbs v s objects obj s tentoonstelling exhibition.50 toon display.72 schilderij painting.47 expositie exposition.49 omvat cover.63 werk work.46 galerie gallery.36 bevat contain.18 tekening drawing.36 collectie collection.29 presenteer present.17 foto picture.33 museum museum.27 laat let.07 sculptuur sculpture.25 oeuvre oeuvre.22 koop buy.07 aquarel aquarelle.20 Kunsthal.19 bezit own.06 object object.19 kunstenaar artist.15 zie see.05 beeld statue.12 dat that.12 koop aan acquire.05 overzicht overview.12 hij he.10 in huis heb own.04 portret portrait.11

71 Quality count 44 dimensions contain similar, framelike semantics 43 dimensions contain less clear-cut semantics single verbs account for one dimension verb senses are mixed up 13 dimensions based on syntax rather than semantics fixed expressions pronomina

72 Evaluation: methodology pseudo-disambiguation task to test generalization capacity (standard automatic evaluation for selectional preferences) s v o s o jongere drink bier coalitie aandeel youngster drink beer coalition share werkgever riskeer boete doel kopzorg employer risk fine goal worry directeur zwaai scepter informateur vodka manager sway sceptre informer wodka 10-fold cross validation (± 300,000 co-occurrences)

73 Evaluation: models Evaluation of 4 different models 2 matrix models 1k verbs (10k subjects + 10k direct objects) singular value decomposition (R) non-negative matrix factorization (R 0 ) 2 tensor models 1k verbs 10k subjects 10k direct objects parallel factor analysis (R) non-negative tensor factorization (R 0 )

74 Evaluation: results dimensions 50 (%) 100 (%) 300 (%) svd ± ± ± 1.01 nmf ± ± ± 0.63 parafac ± ± ± 0.76 ntf ± ± ± 0.16

75 Conclusion novel method able to investigate three-way co-occurrences capable of automatically inducing selectional preferences three-way methods improve on two-way methods non-negativity constraint improves on unconstrained models non-negative tensor factorization outperforms other models

76 Similarity evaluation Wordnet-based similarity Compare results to Dutch cornetto database Two similarity measures: path length: Wu & Palmer similarity measure information theoretic: Lin s similarity measure Nouns close in the hierarchy are tightly similar Pairwise similarity for k similar words Test set of ± 5000 nouns

77 Similarity evaluation Wordnet-based similarity wu & palmer s similarity lin s similarity model k = 1 k = 3 k = 5 k = 1 k = 3 k = 5 document window (w=par) window (w=2) syntax baseline syntax > window (w=2) window (w=par) > document

78 Similarity evaluation Wordnet-based similarity Syntax (with pmi) best Closely followed by small window (with pmi) Large window and document perform much worse dimensionality reduction only helps to improve document-based (a little)

79 Similarity evaluation Cluster quality Compare output of clustering algorithm to gold standard classification Two cluster tasks (esslli 2008 workshop s shared task) concrete noun categorization (44 nouns) 2-way: natural, artefact 3-way: animal, vegetable, artefact 6-way: bird, groundanimal, fruittree, green, tool, vehicle abstract/concrete noun discrimination (30 nouns) 2-way: hi, lo Evaluation measures: Entropy: distribution of classes within cluster (small = good) Purity: ratio of largest class present in cluster (large = good)

80 Similarity evaluation Cluster quality 2-way 3-way 6-way model ent pur ent pur ent pur document window (w=par) window (w=2) syntax-based syntax > window (w=2) document > window (w=par)

81 Similarity evaluation Cluster quality Same tendencies as wordnet-based similarity model with large window seems to extract topically related clusters: aardappel potato, ananas pineapple, banaan banana, champignon mushroom, fles bottle, kers cherry, ketel kettle, kip chicken, kom bowl, lepel spoon, peer pear, sla lettuce, ui onion Similar result for abstract/concrete noun discrimination

82 Similarity evaluation Domain coherence Coherence of semantic domain tags (available in cornetto) particular areas of human knowledge (politics, medicine, sports) topical similarity Ratio of most frequent domain tag (also in tagset of target word) over top 10 similar words Same test set of ± 5000 nouns

83 Similarity evaluation Domain coherence model sim topic document.394 window (w=par).399 window (w=2).414 syntax.441 baseline.048 syntax > window (w=2) = window(w=par) = document

84 Similarity evaluation Domain coherence Syntax still scores best Other models do not perform much worse No real difference between small window and large window large window and document do not extract tight similarity, but they do grasp topical similarity