Introduction to Information Retrieval

Transcription

1 Introduction to Information Retrieval IIR 6&7: Vector Space Model Hinrich Schütze Institute for Natural Language Processing, University of Stuttgart Schütze: Vector space model 1 / 37

2 Models and Methods 1 Boolean model and its limitations (30) 2 Vector space model (30) 3 Probabilistic models (30) 4 Language model-based retrieval (30) 5 Latent semantic indexing (30) 6 Learning to rank (30) Schütze: Vector space model 3 / 37

3 Take-away Schütze: Vector space model 4 / 37

4 Take-away tf-idf weighting: Quick review of tf-idf weighting Schütze: Vector space model 4 / 37

5 Take-away tf-idf weighting: Quick review of tf-idf weighting Vector space model represents queries and documents in a high-dimensional space. Schütze: Vector space model 4 / 37

6 Take-away tf-idf weighting: Quick review of tf-idf weighting Vector space model represents queries and documents in a high-dimensional space. Pivot normalization (or pivoted document length normalization ): alternative to cosine normalization that removes a bias inherent in standard length normalization Schütze: Vector space model 4 / 37

7 Outline 1 tf-idf weighting 2 Vector space model 3 Pivot length normalization Schütze: Vector space model 5 / 37

8 Binary incidence matrix Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is represented as a binary vector {0,1} V. Schütze: Vector space model 6 / 37

9 Binary incidence matrix Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is represented as a binary vector {0,1} V. Schütze: Vector space model 6 / 37

10 Count matrix Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is now represented as a count vector N V. Schütze: Vector space model 7 / 37

11 Count matrix Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is now represented as a count vector N V. Schütze: Vector space model 7 / 37

12 Term frequency tf The term frequency tf t,d of term t in document d is defined as the number of times that t occurs in d. Schütze: Vector space model 8 / 37

13 Term frequency tf The term frequency tf t,d of term t in document d is defined as the number of times that t occurs in d. We want to rank documents according to query-document matching scores and use tf as a component in these matching scores. Schütze: Vector space model 8 / 37

14 Term frequency tf The term frequency tf t,d of term t in document d is defined as the number of times that t occurs in d. We want to rank documents according to query-document matching scores and use tf as a component in these matching scores. But how? Schütze: Vector space model 8 / 37

15 Term frequency tf The term frequency tf t,d of term t in document d is defined as the number of times that t occurs in d. We want to rank documents according to query-document matching scores and use tf as a component in these matching scores. But how? Raw term frequency is not what we want because: Schütze: Vector space model 8 / 37

16 Term frequency tf The term frequency tf t,d of term t in document d is defined as the number of times that t occurs in d. We want to rank documents according to query-document matching scores and use tf as a component in these matching scores. But how? Raw term frequency is not what we want because: A document with tf = 10 occurrences of the term is more relevant than a document with tf = 1 occurrence of the term. Schütze: Vector space model 8 / 37

17 Term frequency tf The term frequency tf t,d of term t in document d is defined as the number of times that t occurs in d. We want to rank documents according to query-document matching scores and use tf as a component in these matching scores. But how? Raw term frequency is not what we want because: A document with tf = 10 occurrences of the term is more relevant than a document with tf = 1 occurrence of the term. But not 10 times more relevant. Schütze: Vector space model 8 / 37

18 Term frequency tf The term frequency tf t,d of term t in document d is defined as the number of times that t occurs in d. We want to rank documents according to query-document matching scores and use tf as a component in these matching scores. But how? Raw term frequency is not what we want because: A document with tf = 10 occurrences of the term is more relevant than a document with tf = 1 occurrence of the term. But not 10 times more relevant. Relevance does not increase proportionally with term frequency. Schütze: Vector space model 8 / 37

19 Instead of raw frequency: Log frequency weighting Schütze: Vector space model 9 / 37

20 Instead of raw frequency: Log frequency weighting The log frequency weight of term t in d is defined as follows { 1 + log10 tf w t,d = t,d if tf t,d > 0 0 otherwise Schütze: Vector space model 9 / 37

21 Instead of raw frequency: Log frequency weighting The log frequency weight of term t in d is defined as follows { 1 + log10 tf w t,d = t,d if tf t,d > 0 0 otherwise tf t,d w t,d : 0 0, 1 1, 2 1.3, 10 2, , etc. Schütze: Vector space model 9 / 37

22 Instead of raw frequency: Log frequency weighting The log frequency weight of term t in d is defined as follows { 1 + log10 tf w t,d = t,d if tf t,d > 0 0 otherwise tf t,d w t,d : 0 0, 1 1, 2 1.3, 10 2, , etc. Matching score for a document-query pair: sum over terms t in both q and d: tf-matching-score(q,d) = t q d (1 + log tf t,d) Schütze: Vector space model 9 / 37

23 Frequency in document vs. frequency in collection In addition, to term frequency (the frequency of the term in the document)... Schütze: Vector space model 10 / 37

24 Frequency in document vs. frequency in collection In addition, to term frequency (the frequency of the term in the document)......we also want to use the frequency of the term in the collection for weighting and ranking. Schütze: Vector space model 10 / 37

25 idf weight Schütze: Vector space model 11 / 37

26 idf weight df t is the document frequency, the number of documents that t occurs in. Schütze: Vector space model 11 / 37

27 idf weight df t is the document frequency, the number of documents that t occurs in. df t is an inverse measure of the informativeness of term t. Schütze: Vector space model 11 / 37

28 idf weight df t is the document frequency, the number of documents that t occurs in. df t is an inverse measure of the informativeness of term t. Inverse document frequency, idf t, is a direct measure of the informativeness of the term. Schütze: Vector space model 11 / 37

29 idf weight df t is the document frequency, the number of documents that t occurs in. df t is an inverse measure of the informativeness of term t. Inverse document frequency, idf t, is a direct measure of the informativeness of the term. The idf weight of term t is defined as follows: idf t = log 10 N df t (N is the number of documents in the collection.) Schütze: Vector space model 11 / 37

30 idf weight df t is the document frequency, the number of documents that t occurs in. df t is an inverse measure of the informativeness of term t. Inverse document frequency, idf t, is a direct measure of the informativeness of the term. The idf weight of term t is defined as follows: idf t = log 10 N df t (N is the number of documents in the collection.) [log N/df t ] instead of [N/df t ] to dampen the effect of idf Schütze: Vector space model 11 / 37

31 Examples for idf Schütze: Vector space model 12 / 37

32 Examples for idf idf t = log 10 1,000,000 df t term df t idf t calpurnia 1 6 animal sunday fly 10,000 2 under 100,000 1 the 1,000,000 0 Schütze: Vector space model 12 / 37

33 Effect of idf on ranking Schütze: Vector space model 13 / 37

34 Effect of idf on ranking idf gives high weights to rare terms like arachnocentric. Schütze: Vector space model 13 / 37

35 Effect of idf on ranking idf gives high weights to rare terms like arachnocentric. idf gives low weights to frequent words like good, increase, and line. Schütze: Vector space model 13 / 37

36 Effect of idf on ranking idf gives high weights to rare terms like arachnocentric. idf gives low weights to frequent words like good, increase, and line. idf affects the ranking of documents for queries with at least two terms. Schütze: Vector space model 13 / 37

37 Effect of idf on ranking idf gives high weights to rare terms like arachnocentric. idf gives low weights to frequent words like good, increase, and line. idf affects the ranking of documents for queries with at least two terms. For example, in the query arachnocentric line, idf weighting increases the relative weight of arachnocentric and decreases the relative weight of line. Schütze: Vector space model 13 / 37

38 Effect of idf on ranking idf gives high weights to rare terms like arachnocentric. idf gives low weights to frequent words like good, increase, and line. idf affects the ranking of documents for queries with at least two terms. For example, in the query arachnocentric line, idf weighting increases the relative weight of arachnocentric and decreases the relative weight of line. idf has little effect on ranking for one-term queries. Schütze: Vector space model 13 / 37

39 Summary: tf-idf weighting Schütze: Vector space model 14 / 37

40 Summary: tf-idf weighting Assign a tf-idf weight for each term t in each document d: w t,d = (1 + log tf t,d ) log N df t Schütze: Vector space model 14 / 37

41 Summary: tf-idf weighting Assign a tf-idf weight for each term t in each document d: w t,d = (1 + log tf t,d ) log N df t The tf-idf weight... Schütze: Vector space model 14 / 37

42 Summary: tf-idf weighting Assign a tf-idf weight for each term t in each document d: w t,d = (1 + log tf t,d ) log df N t The tf-idf weight increases with the number of occurrences within a document. (term frequency component) Schütze: Vector space model 14 / 37

43 Summary: tf-idf weighting Assign a tf-idf weight for each term t in each document d: w t,d = (1 + log tf t,d ) log df N t The tf-idf weight increases with the number of occurrences within a document. (term frequency component)... increases with the rarity of the term in the collection. (inverse document frequency component) Schütze: Vector space model 14 / 37

44 Outline 1 tf-idf weighting 2 Vector space model 3 Pivot length normalization Schütze: Vector space model 15 / 37

45 Binary incidence matrix Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is represented as a binary vector {0,1} V. Schütze: Vector space model 16 / 37

46 Count matrix Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is now represented as a count vector N V. Schütze: Vector space model 17 / 37

47 Binary count weight matrix Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is now represented as a real-valued vector of tf-idf weights R V. Schütze: Vector space model 18 / 37

48 Binary count weight matrix Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is now represented as a real-valued vector of tf-idf weights R V. Schütze: Vector space model 18 / 37

49 Documents as vectors Each document is now represented as a real-valued vector of tf-idf weights R V. Schütze: Vector space model 19 / 37

50 Documents as vectors Each document is now represented as a real-valued vector of tf-idf weights R V. So we have a V -dimensional real-valued vector space. Schütze: Vector space model 19 / 37

51 Documents as vectors Each document is now represented as a real-valued vector of tf-idf weights R V. So we have a V -dimensional real-valued vector space. Terms are axes of the space. Schütze: Vector space model 19 / 37

52 Documents as vectors Each document is now represented as a real-valued vector of tf-idf weights R V. So we have a V -dimensional real-valued vector space. Terms are axes of the space. Documents are points or vectors in this space. Schütze: Vector space model 19 / 37

53 Documents as vectors Each document is now represented as a real-valued vector of tf-idf weights R V. So we have a V -dimensional real-valued vector space. Terms are axes of the space. Documents are points or vectors in this space. Very high-dimensional: tens of millions of dimensions when you apply this to web search engines Schütze: Vector space model 19 / 37

54 Documents as vectors Each document is now represented as a real-valued vector of tf-idf weights R V. So we have a V -dimensional real-valued vector space. Terms are axes of the space. Documents are points or vectors in this space. Very high-dimensional: tens of millions of dimensions when you apply this to web search engines Each vector is very sparse - most entries are zero. Schütze: Vector space model 19 / 37

55 Queries as vectors Key idea 1: do the same for queries: represent them as vectors in the high-dimensional space Schütze: Vector space model 20 / 37

56 Queries as vectors Key idea 1: do the same for queries: represent them as vectors in the high-dimensional space Key idea 2: Rank documents according to their proximity to the query Schütze: Vector space model 20 / 37

57 Queries as vectors Key idea 1: do the same for queries: represent them as vectors in the high-dimensional space Key idea 2: Rank documents according to their proximity to the query proximity = similarity Schütze: Vector space model 20 / 37

58 Queries as vectors Key idea 1: do the same for queries: represent them as vectors in the high-dimensional space Key idea 2: Rank documents according to their proximity to the query proximity = similarity proximity negative distance Schütze: Vector space model 20 / 37

59 Queries as vectors Key idea 1: do the same for queries: represent them as vectors in the high-dimensional space Key idea 2: Rank documents according to their proximity to the query proximity = similarity proximity negative distance Recall: We re doing this because we want to get away from the you re-either-in-or-out, feast-or-famine Boolean model. Schütze: Vector space model 20 / 37

60 Queries as vectors Key idea 1: do the same for queries: represent them as vectors in the high-dimensional space Key idea 2: Rank documents according to their proximity to the query proximity = similarity proximity negative distance Recall: We re doing this because we want to get away from the you re-either-in-or-out, feast-or-famine Boolean model. Instead: rank relevant documents higher than nonrelevant documents Schütze: Vector space model 20 / 37

61 How do we formalize vector space similarity? Schütze: Vector space model 21 / 37

62 How do we formalize vector space similarity? First cut: (negative) distance between two points Schütze: Vector space model 21 / 37

63 How do we formalize vector space similarity? First cut: (negative) distance between two points ( = distance between the end points of the two vectors) Schütze: Vector space model 21 / 37

64 How do we formalize vector space similarity? First cut: (negative) distance between two points ( = distance between the end points of the two vectors) Euclidean distance? Schütze: Vector space model 21 / 37

65 How do we formalize vector space similarity? First cut: (negative) distance between two points ( = distance between the end points of the two vectors) Euclidean distance? Euclidean distance is a bad idea... Schütze: Vector space model 21 / 37

66 How do we formalize vector space similarity? First cut: (negative) distance between two points ( = distance between the end points of the two vectors) Euclidean distance? Euclidean distance is a bad idea......because Euclidean distance is large for vectors of different lengths. Schütze: Vector space model 21 / 37

67 Why distance is a bad idea Schütze: Vector space model 22 / 37

68 Why distance is a bad idea poor 1 d 1 :Ranks of starving poets swell d 2 :Rich poor gap grows q:[rich poor] 0 d 3 :Record baseball salaries in 2010 rich 0 1 The Euclidean distance of q and d 2 is large although the distribution of terms in the query q and the distribution of terms in the document d 2 are very similar. Schütze: Vector space model 22 / 37

69 Use angle instead of distance Schütze: Vector space model 23 / 37

70 Use angle instead of distance Rank documents according to angle with query Schütze: Vector space model 23 / 37

71 Use angle instead of distance Rank documents according to angle with query The following two notions are equivalent. Schütze: Vector space model 23 / 37

72 Use angle instead of distance Rank documents according to angle with query The following two notions are equivalent. Rank documents according to the angle between query and document in decreasing order Schütze: Vector space model 23 / 37

73 Use angle instead of distance Rank documents according to angle with query The following two notions are equivalent. Rank documents according to the angle between query and document in decreasing order Rank documents according to cosine(query,document) in increasing order Schütze: Vector space model 23 / 37

74 Use angle instead of distance Rank documents according to angle with query The following two notions are equivalent. Rank documents according to the angle between query and document in decreasing order Rank documents according to cosine(query,document) in increasing order Cosine is a monotonically decreasing function of the angle for the interval [0,180 ] Schütze: Vector space model 23 / 37

75 Use angle instead of distance Rank documents according to angle with query The following two notions are equivalent. Rank documents according to the angle between query and document in decreasing order Rank documents according to cosine(query,document) in increasing order Cosine is a monotonically decreasing function of the angle for the interval [0,180 ] do ranking according to cosine Schütze: Vector space model 23 / 37

76 Cosine similarity between query and document Schütze: Vector space model 24 / 37

77 Cosine similarity between query and document cos( q, d) = sim( q, d) = q q d d = V i=1 q i q di d Schütze: Vector space model 24 / 37

78 Cosine similarity between query and document cos( q, d) = sim( q, d) = q q d d = q i is the tf-idf weight of term i in the query. V i=1 q i q di d Schütze: Vector space model 24 / 37

79 Cosine similarity between query and document cos( q, d) = sim( q, d) = q q d d = q i is the tf-idf weight of term i in the query. V i=1 d i is the tf-idf weight of term i in the document. q i q di d Schütze: Vector space model 24 / 37

80 Cosine similarity between query and document cos( q, d) = sim( q, d) = q q d d = q i is the tf-idf weight of term i in the query. V i=1 d i is the tf-idf weight of term i in the document. q and d are the lengths of q and d. q i q di d Schütze: Vector space model 24 / 37

81 Cosine similarity between query and document cos( q, d) = sim( q, d) = q q d d = q i is the tf-idf weight of term i in the query. V i=1 d i is the tf-idf weight of term i in the document. q and d are the lengths of q and d. q i q di d This is the cosine similarity of q and d....or, equivalently, the cosine of the angle between q and d. Schütze: Vector space model 24 / 37

82 Cosine similarity between query and document cos( q, d) = sim( q, d) = q q d d = q i is the tf-idf weight of term i in the query. V i=1 d i is the tf-idf weight of term i in the document. q and d are the lengths of q and d. q i q di d This is the cosine similarity of q and d....or, equivalently, the cosine of the angle between q and d. cosine similarity = dot product of length-normalized vectors Schütze: Vector space model 24 / 37

83 Cosine similarity illustrated Schütze: Vector space model 25 / 37

84 Cosine similarity illustrated poor 1 v(d 1 ) v(q) v(d 2 ) θ v(d 3 ) rich Schütze: Vector space model 25 / 37

85 Components of tf-idf weighting Term frequency Document frequency Normalization n (natural) tf t,d n (no) 1 n (none) 1 l (logarithm) 1 + log(tf t,d ) t (idf) log N dft c (cosine) 1 w 2 1 +w w2 M a (augmented) b (boolean) tft,d maxt(tft,d) { 1 if tft,d > 0 0 otherwise p (prob idf) max{0,log N dft } u (pivoted dft unique) 1/u b (byte size) 1/CharLength α, α < 1 L (log ave) 1+log(tft,d) 1+log(avet d(tft,d)) Schütze: Vector space model 26 / 37

86 Components of tf-idf weighting Term frequency Document frequency Normalization n (natural) tf t,d n (no) 1 n (none) 1 l (logarithm) 1 + log(tf t,d ) t (idf) log N dft c (cosine) 1 w 2 1 +w w2 M a (augmented) b (boolean) tft,d maxt(tft,d) { 1 if tft,d > 0 0 otherwise p (prob idf) max{0,log N dft } u (pivoted dft unique) 1/u b (byte size) 1/CharLength α, α < 1 L (log ave) 1+log(tft,d) 1+log(avet d(tft,d)) Best known combination of weighting options Schütze: Vector space model 26 / 37

87 tf-idf example Schütze: Vector space model 27 / 37

88 tf-idf example We often use different weightings for queries and documents. Schütze: Vector space model 27 / 37

89 tf-idf example We often use different weightings for queries and documents. Notation: ddd.qqq Schütze: Vector space model 27 / 37

90 tf-idf example We often use different weightings for queries and documents. Notation: ddd.qqq Example: lnc.ltn Schütze: Vector space model 27 / 37

91 tf-idf example We often use different weightings for queries and documents. Notation: ddd.qqq Example: lnc.ltn document: logarithmic tf, no df weighting, cosine normalization Schütze: Vector space model 27 / 37

92 tf-idf example We often use different weightings for queries and documents. Notation: ddd.qqq Example: lnc.ltn document: logarithmic tf, no df weighting, cosine normalization query: logarithmic tf, idf, no normalization Schütze: Vector space model 27 / 37

93 tf-idf example We often use different weightings for queries and documents. Notation: ddd.qqq Example: lnc.ltn document: logarithmic tf, no df weighting, cosine normalization query: logarithmic tf, idf, no normalization Isn t it bad to not idf-weight the document? Schütze: Vector space model 27 / 37

94 tf-idf example We often use different weightings for queries and documents. Notation: ddd.qqq Example: lnc.ltn document: logarithmic tf, no df weighting, cosine normalization query: logarithmic tf, idf, no normalization Isn t it bad to not idf-weight the document? Example query: best car insurance Schütze: Vector space model 27 / 37

95 tf-idf example We often use different weightings for queries and documents. Notation: ddd.qqq Example: lnc.ltn document: logarithmic tf, no df weighting, cosine normalization query: logarithmic tf, idf, no normalization Isn t it bad to not idf-weight the document? Example query: best car insurance Example document: car insurance auto insurance Schütze: Vector space model 27 / 37

96 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto best car insurance Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Schütze: Vector space model 28 / 37

97 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto 0 best 1 car 1 insurance 1 Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Schütze: Vector space model 28 / 37

98 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto 0 1 best 1 0 car 1 1 insurance 1 2 Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Schütze: Vector space model 28 / 37

99 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto best car insurance Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Schütze: Vector space model 28 / 37

100 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto best car insurance Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Schütze: Vector space model 28 / 37

101 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto best car insurance Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Schütze: Vector space model 28 / 37

102 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto best car insurance Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Schütze: Vector space model 28 / 37

103 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto best car insurance Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Schütze: Vector space model 28 / 37

104 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto best car insurance Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Schütze: Vector space model 28 / 37

105 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto best car insurance Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Schütze: Vector space model 28 / 37

106 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto best car insurance Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight / / Schütze: Vector space model 28 / 37

107 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto best car insurance Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Schütze: Vector space model 28 / 37

108 tf-idf example: lnc.ltn Query: best car insurance. Document: car insurance auto insurance. word query document product tf-raw tf-wght df idf weight tf-raw tf-wght weight n lized auto best car insurance Key to columns: tf-raw: raw (unweighted) term frequency, tf-wght: logarithmically weighted term frequency, df: document frequency, idf: inverse document frequency, weight: the final weight of the term in the query or document, n lized: document weights after cosine normalization, product: the product of final query weight and final document weight Final similarity score between query and document: i w qi w di = = 3.08 Schütze: Vector space model 28 / 37

109 Outline 1 tf-idf weighting 2 Vector space model 3 Pivot length normalization Schütze: Vector space model 29 / 37

110 A problem for cosine normalization Schütze: Vector space model 30 / 37

111 A problem for cosine normalization Query q: anti-doping rules Beijing 2008 olympics Schütze: Vector space model 30 / 37

112 A problem for cosine normalization Query q: anti-doping rules Beijing 2008 olympics Compare three documents Schütze: Vector space model 30 / 37

113 A problem for cosine normalization Query q: anti-doping rules Beijing 2008 olympics Compare three documents d 1 : a short document on anti-doping rules at 2008 Olympics Schütze: Vector space model 30 / 37

114 A problem for cosine normalization Query q: anti-doping rules Beijing 2008 olympics Compare three documents d 1 : a short document on anti-doping rules at 2008 Olympics d 2 : a long document that consists of a copy of d 1 and 5 other news stories, all on topics different from Olympics/anti-doping Schütze: Vector space model 30 / 37

115 A problem for cosine normalization Query q: anti-doping rules Beijing 2008 olympics Compare three documents d 1 : a short document on anti-doping rules at 2008 Olympics d 2 : a long document that consists of a copy of d 1 and 5 other news stories, all on topics different from Olympics/anti-doping d 3 : a short document on anti-doping rules at the 2004 Athens Olympics Schütze: Vector space model 30 / 37

116 A problem for cosine normalization Query q: anti-doping rules Beijing 2008 olympics Compare three documents d 1 : a short document on anti-doping rules at 2008 Olympics d 2 : a long document that consists of a copy of d 1 and 5 other news stories, all on topics different from Olympics/anti-doping d 3 : a short document on anti-doping rules at the 2004 Athens Olympics What ranking do we expect in the vector space model? Schütze: Vector space model 30 / 37

117 Pivot normalization Cosine normalization produces weights that are too large for short documents and too small for long documents (on average). Schütze: Vector space model 31 / 37

118 Pivot normalization Cosine normalization produces weights that are too large for short documents and too small for long documents (on average). Adjust cosine normalization by linear adjustment: turning the average normalization on the pivot Schütze: Vector space model 31 / 37

119 Pivot normalization Cosine normalization produces weights that are too large for short documents and too small for long documents (on average). Adjust cosine normalization by linear adjustment: turning the average normalization on the pivot Effect: Similarities of short documents with query decrease; similarities of long documents with query increase. Schütze: Vector space model 31 / 37

120 Pivot normalization Cosine normalization produces weights that are too large for short documents and too small for long documents (on average). Adjust cosine normalization by linear adjustment: turning the average normalization on the pivot Effect: Similarities of short documents with query decrease; similarities of long documents with query increase. This removes the unfair advantage that short documents have. Schütze: Vector space model 31 / 37

121 Pivot normalization Cosine normalization produces weights that are too large for short documents and too small for long documents (on average). Adjust cosine normalization by linear adjustment: turning the average normalization on the pivot Effect: Similarities of short documents with query decrease; similarities of long documents with query increase. This removes the unfair advantage that short documents have. Singhal s study is also interesting from the point of view of methodology. Schütze: Vector space model 31 / 37

122 Predicted and true probability of relevance Schütze: Vector space model 32 / 37

123 Predicted and true probability of relevance source: Lillian Lee Schütze: Vector space model 32 / 37

124 Pivot normalization Schütze: Vector space model 33 / 37

125 Pivot normalization source: Lillian Lee Schütze: Vector space model 33 / 37

126 Pivoted normalization: Amit Singhal s experiments Schütze: Vector space model 34 / 37

127 Pivoted normalization: Amit Singhal s experiments (relevant documents retrieved and (change in) average precision) Schütze: Vector space model 34 / 37

128 Summary: Ranked retrieval in the vector space model Schütze: Vector space model 35 / 37

129 Summary: Ranked retrieval in the vector space model Represent each document as a weighted tf-idf vector Schütze: Vector space model 35 / 37

130 Summary: Ranked retrieval in the vector space model Represent each document as a weighted tf-idf vector Represent the query as a weighted tf-idf vector Schütze: Vector space model 35 / 37

131 Summary: Ranked retrieval in the vector space model Represent each document as a weighted tf-idf vector Represent the query as a weighted tf-idf vector Compute the cosine similarity between the query vector and each document vector Schütze: Vector space model 35 / 37

132 Summary: Ranked retrieval in the vector space model Represent each document as a weighted tf-idf vector Represent the query as a weighted tf-idf vector Compute the cosine similarity between the query vector and each document vector Alternatively, use pivot normalization Schütze: Vector space model 35 / 37

133 Summary: Ranked retrieval in the vector space model Represent each document as a weighted tf-idf vector Represent the query as a weighted tf-idf vector Compute the cosine similarity between the query vector and each document vector Alternatively, use pivot normalization Rank documents with respect to the query Schütze: Vector space model 35 / 37

134 Summary: Ranked retrieval in the vector space model Represent each document as a weighted tf-idf vector Represent the query as a weighted tf-idf vector Compute the cosine similarity between the query vector and each document vector Alternatively, use pivot normalization Rank documents with respect to the query Return the top K (e.g., K = 10) to the user Schütze: Vector space model 35 / 37

135 Take-away tf-idf weighting: Quick review of tf-idf weighting Vector space model represents queries and documents in a high-dimensional space. Pivot normalization (or pivoted document length normalization ): alternative to cosine normalization that removes a bias inherent in standard length normalization Schütze: Vector space model 36 / 37

136 Resources Chapters 6 and 7 of Introduction to Information Retrieval Resources at Gerard Salton (main proponent of vector space model in 70s, 80s, 90s) Exploring the similarity space (Moffat and Zobel, 2005) Pivot normalization (original paper) Schütze: Vector space model 37 / 37