Time-Based Language Models

Transcription

1 Tie-Based Language Models Xiaoyan Li and W. Bruce Croft Center for Intelligent Inforation Retrieval epartent of Coputer Science University of Massachusetts, Aherst, MA 3 {xiaoyan,croft}@cs.uass.edu ABSTRACT We explore the relationship between tie and relevance using TREC ad-hoc queries. A type of query is identified that favors very recent docuents. We propose a tie-based language odel approach to retrieval for these queries. We show how tie can be incorporated into both query-likelihood odels and relevance odels. We carried out experients to copare tie-based language odels to heuristic techniques for incorporating docuent recency in the ranking. Our results show that tiebased odels perfor as well as or better than the best of the heuristic techniques. KEYWORS Inforation retrieval, language odels, relevance odels, tiebased language odels, recency queries. INTROUCTION The task of inforation retrieval is to retrieve relevant docuents that satisfy the user s inforation need. evance is an abstract easure of how well a docuent satisfies the user's inforation need, which is approxiated by a query. In the process of approxiation, a tie-related inforation need is usually not captured by the query. For exaple, an old docuent, which is topically relevant to the query, ay not satisfy the inforation need if the user is only interested in ore recent docuents. Many news-related queries would fall into this category. It is also possible that a recent docuent that appears to be topically relevant ay not satisfy the user s inforation need if the user is only interested in docuents within a specific period in the past. For exaple, the query star wars could have ost of the relevant docuents in the Reagan era rather than in recent docuents. Most docuent retrieval systes built for the corporate environent recognize the iportance of tie and have provided, for any years, default rankings based on recency as well as the ability to specify a tie period as a query attribute or field. The proble with these systes is that they are either based on a Boolean retrieval odel or the tie attribute is cobined in a heuristic anner with the docuent scores to produce a final ranking. In this paper, we introduce the tie-based language odel approach that incorporates tie as part of the retrieval odel. Tie-based language odels are a siple extension of the language odel approaches to retrieval that have been developed over the past few years (e.g. [-6]). Instead of assuing unifor prior probabilities in these retrieval odels, we assign docuent priors based on creation dates. In the next section, we explore the relationship between tie and relevance on TREC ad-hoc title queries, and identify recency queries that favor very recent docuents for evaluating the proposed odels. Section 3 describes the tie-based language odel approaches to retrieval. Section 4 gives the experiental design and results. The experiental results show that tie-based language odels generally outperfor heuristic techniques. ated research is discussed in section 5, and section 6 discusses future research directions. 2. TIME AN RELEVANCE In this section, we explore the relationship between tie and relevance based on an analysis of TREC ad-hoc queries. The first part shows the average distribution over tie for the TREC relevant docuents. The second part highlights the differences between individual queries with respect to tie sensitivity. 2. Tie and evance in TREC Figure 2. is an exaple of the tie distribution of relevance judgents for TREC queries (in this case, queries 25-3). The x axis represents tie in onths (in the past) and the y axis represents the percentage of total relevant docuents. The origin corresponds to the ost recent date in all the TREC collections. These averages are affected by a nuber of factors, such as when the collections were introduced, and which collections were used in a given year, but soe trends can be observed. In the distribution shown, relevant docuents are distributed fairly evenly across the tie line, but are ore concentrated in the older docuents. There is even a short period where there were no relevant docuents, due to gaps in the source collections.

2 Figure 2.: istribution over tie for relevant docuents (queries 25-3) Figure 2.3: Query 56 - evant docuents ostly in the past. 2.2 Exaples of ifferent Types of Queries. Individual queries can show uch ore tie sensitivity than the averages. As we entioned previously, there are two ain types of queries that do not have a unifor distribution of relevant docuents over tie (there are actually any types of distributions but these are ore coon). The first type of query favors very recent docuents and the other has ore relevant docuents within a specific period in the past. Query 3 is an exaple of the first type of query. (See figure 2.2). Query 56 is an exaple of the second type of query, which has ore relevant docuents within a particular period in the past. (See figure 2.3) Query 65 is an exaple of a query that has a ore unifor distribution of relevant docuents along the tie line. (See figure 2.4). This group is the ost nuerous, but there are still a significant nuber of exaples of the first two types. In this paper we are ore interested in the first type of queries: recency queries. These queries favor very recent docuents. In the TREC queries 3-4, we anually identified 36 recency queries that are used in the experients described in section 4. It is iportant to ephasize that the distribution of relevant docuents over tie is substantially ore biased than the background distribution of the collection. Figure 2.2: Query 3 - A recency query. Figure 2.4: Query 65 - More unifor distribution. 3. LANGUAGE MOELS FOR RETRIEVAL 3. Query Likelihood Models Language odeling fraeworks were introduced to inforation retrieval by Ponte and Croft [], followed by soe variations [2,3,4,5] that adopted a siilar fraework. In the language odeling fraework, there are basically three approaches to ranking docuents: the query likelihood odel, the docuent likelihood odel and coparing query and docuent language odels directly. In the siplest case, the posterior probability of a docuent given in (3.) is used to rank the docuents in the collection. d / q) q / d) d) (3.) The prior probability of the docuent d) is usually assued to be unifor and is ignored for ranking. Ponte and Croft treat the query Q as a binary vector over the entire vocabulary and use (3.2) for estiating of the probability of generating query text (the notation M is used to indicate that the query is generated by a docuent language odel). w Q P ( Q / M ) = w / M ) ( M ) w Q (3.2) Song and Croft [2], Hiestra [3], and Miller et al [6] treat the query Q as a sequence of independent words instead of a binary

3 vector and use (3.3) for query likelihood ( qw is the nuber of ties the word w occurs in the query). w P ( Q / M ) = w / M ) (3.3) w In the present study, the forula specified in equation (3.3) is used as a baseline in the experients. 3.2 evance odels Lavrenko and Croft [5] incorporate relevance feedback and query expansion into language odeling fraeworks. They proposed a technique for estiating a relevance odel based on the query. The relevance odel, R), is estiated using a joint probability of observing the word w together with query words q q,...,, 2 q. w, q,..., q) w, q,..., q) R) Q) = = (3.4) q,..., q ) v, q,..., q ) q vocabulary Lavrenko and Croft describe two ethods of estiating the joint probability. The two ethods differ in the independence assuptions that are being ade. The first ethod assues that w was sapled in the sae way as the query words. The second ethod assues that w and the query words were sapled using two different echaniss. There is no significant difference on perforance between these two ethods and the first ethod was reported ore efficient. Therefore, we use the first ethod in this paper. If we assue that w and q, q2,..., q are utually independent once we pick a distribution M, then we get: w, q,..., q) = M) M) qi / M) (3.5) M Μ i= Here M) denotes soe prior probability which is kept unifor over all distributions M. The KL divergence between the relevance odel and a docuent odel, which is given in equation (3.6), can be used to rank docuents. ocuents with saller divergence are considered ore relevant. w / R) KL( R M d ) = w / R) log (3.6) w / M ) w In the present study, we use equation (3.6) for the baseline relevance odel in the experients. 3.3 Tie-Based Language Models The study of the relationship between tie and relevance in section 2 shows that for tie-based queries, docuents with different docuent creation dates/tiestaps ay have different prior probabilities for relevance. Therefore, we propose to replace d) in equation (3.) and M) in equation (3.5) with soe probability dependent on docuents date T, say p d / T ) or T d ( d p ( M / ). This gives us the tie-based language odels: p d / q) q / d) d / T ) (3.7) ( d and w, q,..., q) = M / T ) M) qi / M) (3.8) M Μ i= Although p d / T ) in the query likelihood language odel ( d and p ( M / T ) in the relevance odel have soewhat different eanings, we refer to both as p ( / T ) for siplicity. In the case of the relevance odel, the tie-based prior will affect the docuents that are used to construct the odel. When viewed as a for of query expansion, this eans that the expansion will be based on the top-ranked docuents subject to a tie constraint, such as favoring the ost recent docuents. This property could be exploited to change the interpretation of a query in, for exaple, systes with user odels that change over tie. The next challenge is to estiate the probability p ( / T ). We suggest soe siple ethod for estiating this probability for recency queries. For queries where recency is a ajor requireent of a user s inforation need, we used an exponential distribution for prior probability assignent. The prior p ( / T ) is given in equation (3.9). ocuents with a ore recent creation date are assigned higher probability. p λ( ) ( / ) ( ) T C T T = P T = e λ (3.9) Here TC is the ost recent date (in onth) in the whole collection and T is the creation date of a docuent. The training of the paraeters in equation (3.9) and the experiental results are detailed in section EXPERIMENTAL ESIGN AN RESULTS 4. ata The data consists of 36 recency queries fro TREC queries 3-4 over collections fro TREC volues 4 and volue 5. The collection we used is also tie-biased since it has ore docuents in the recent past, which is siilar to the inforation sources of web pages. The data set is then randoly split into two sets: 2 queries are randoly picked for training the paraeters in the tie-based language odels and the heuristic techniques. The other 6 recency queries are used as a test set to test the perforance of different approaches. The specific queries used in these sets are listed in the appendix. 4.2 Baselines We considered three baselines for coparison. The first baseline is retrieval using the query likelihood language odels or relevance-based language odels with unifor priors. The second baseline involves reranking the top N docuents solely by

4 recency, which is deterined by docuent creation date. We reranked the top and 5 docuents respectively and keep the rest of the retrieved docuents unchanged. This technique is used as an option in any retrieval systes. The third baseline is reranking docuents by a linear cobination of the original topicality rank and a recency rank based on creation date. The algorith is as follows: (). Take top retrieved docuents. (2). Copute Score = (-α) R topicality +α R recency (3). Rerank docuents by increasing score. R topicality is the rank of a docuent in ters of the original belief score, i.e. the original rank in the list of retrieved docuents. R recency is the rank of a docuent in ters of recency. The ost recent docuent retrieved in the top docuents will have a value for R recency. 4.3 Experiental design We carried out four sets of training experients and two sets of test experients. The first set of experients was used to deterine the best value of λ in the exponential distribution on tie-based query likelihood language odels. The second set of experients was to deterine the best value of λ in the exponential distribution on tie-based relevance language odels. The third set of experients was to deterine the best value of α in the linear cobination on docuents retrieved with query likelihood language odels. The forth set of experients was to deterine the best value of α in the linear cobination on docuents retrieved with relevance-based language odels. For each set of training experients, a nuber of different paraeter values were tested. The paraeter value with highest perforance in ters of average precision was chosen as best paraeter value for the experients with the test set. Table shows results with the tie-based relevance odels on the training set. Only three values of λare shown here, although ore were tried. The best value in ters of perforance was.2. This eans that the exponential distribution given in figure 4. is used to assign prior probability for tie-based query likelihood language odels. The best value of λ was. with the tiebased query likelihood odel. Table 2 shows the results using different values of α with the linear cobination and the recency queries in the training set. In this case the value of.4 for α produced the best results. The two sets of test experients use paraeters deterined fro the training experients. The first set of test experients is for the coparison of the tie-based query likelihood odel to the three baselines: query likelihood language odels, reranking solely by recency and linear cobinations. The second set of test experients is for the coparison of the tie-based relevance odel to the three baselines: relevance-based language odels, reranking solely by recency and linear cobinations. The results are shown in Table 3 and Table 4 respectively. iscussion about the results is detailed in section 4.4. Table : Training for Tie-based evance Models RM TB2-. TB2-.2 TB2-.3 (λ*=.2) Avg RM: relevance-based language odel. TB2-a: tie-based relevance odel with λ = a in the exponential distribution. Table 2: Training for Linear Cobination RM LC-.3 LC-.4 LC-.5 (α*=.4) Avg LC-a: Linear Cobination with α = a. 4.4 iscussion Table 3 shows that the tie-based query likelihood odel with its best value of λ, which is learned fro the training process, outperfors the query likelihood language odel, reranking solely by recency with two docuent cut-off levels and the linear cobination with its best value of α. Table 4 shows that the tiebased relevance odel with its best value of λ outperfors the first two baselines but is not as good as the linear cobination with its best α value. However, after a closer look at the training process, we found that the best perforance of the tie-based relevance odel is uch better, a 7.9% increase in ters of average precision, than the best perforance of the linear cobination on training set. See Table and Table 2. Another observation on the test set is that the perforance of the

5 relevance-based language odel is worse than the perforance of the query likelihood language odel. That probably explains why the tie-based relevance odel achieves little iproveent over relevance-based language odel on this test set, and doesn t perfor as well as the linear cobination. For the queries in the training set, on average the relevance-based language odel outperfors the query likelihood language odel and our tiebased relevance odel outperfors the relevance odel and the linear cobination. In both test sets of experients, tie-based language odels substantially outperfor reranking solely by recency at different docuent cut-off levels. Table 3: Coparison of tie-based query likelihood language odels to baselines on test set LM top top5 LC-. TB Avg Table 4: Coparison of tie-based relevance odels to baselines on test set RM top top5 LC-.4 TB Avg RM: relevance odel TB2-b: tie-based relevance odel with λ = b in the exponential distribution LM: query likelihood language odel. topn: Rerank top N ranked docuents solely by recency. TB-.: tie-based query likelihood odels with λ=.. LC-.: Linear Cobination with α=.. Figure 4.: Exponential distribution used for priors Given that on average relevance-based language odels outperfor query likelihood odels significantly [5], it is possible to show that our tie-based relevance odel ay outperfor linear cobination when a large data set is available. Figure 4.2 and figure 4.3 show the sensitivity of the results in ters of average precision to paraeter values in tie-based language odels and linear cobinations respectively on the whole data set. It can be seen that both approaches are very sensitive to this value. The results of tie-based language odels are ore sensitive to λ when λ is in the range of [,.] than the results of linear cobination with α in the sae range. Figure 4.2: Sensitivity of average precision to λ in tie-based language odels

6 In future work, we will develop techniques to autoatically classify tie-based queries and set paraeters. We have also started using these techniques for tie-based question answering. A nuber of questions, such as Who is the prie inister of Australia?, have tie-dependent answers. We are attepting to use the tie-based language odels to change the ranking of answer passages and the subsequent answers that are extracted. Our goal is to have a tie slide bar that would change the answer as it is oved. For this work, we are using extracted dates in addition to docuent dates. 7. ACKNOWLEGMENTS Figure 4.3: Sensitivity of average precision to α in the linear cobination reranking 5. RELATE RESEARCH As entioned previously, the creation date of a docuent has long been recognized as an iportant attribute in coercial IR systes [2, 3]. In ters of research, the role of tie in retrieval has been soewhat neglected, although recency is often entioned in discussions of relevance and utility. There has been work on constructing tielines autoatically fro tie-tagged retrieved docuents as a visualization and discovery tool (e.g. [7, 8]). There has also been research that exploits the teporal aspect of news streas to iprove topic tracking and the detection of novel inforation [9]. Other related work includes efforts to iprove the extraction of tie tags for question answering [] and incorporating prior probabilities into language odels for entry page search []. Unlike the fixed probability learned for each category of web pages, an exponential distribution is used in this paper to replace unifor distribution in both tie-based query likelihood odels and tie-based relevance odels. 6. CONCLUSIONS AN FUTURE WORK In this paper, we studied the relationship between tie and relevance based on TREC ad-hoc title queries. We proposed tiebased language odel fraeworks, which incorporate tie into both query likelihood language odels and relevance-based language odels. Exponential distributions are used to replace the unifor prior probability in these odels. Our epirical results show that, for a particular set of recency queries, tiebased query likelihood language odels outperfors three baselines: query likelihood language odels, reranking solely by recency and linear cobination reranking. The tie-based relevance odel outperfors the relevance-based language odel and reranking solely by recency. The ain contribution of this work is to show that contextual features such as tie constraints can be incorporated into the underlying retrieval odel without resorting to heuristic approaches. This work was supported in part by the Center for Intelligent Inforation Retrieval, in part by SPAWARSYSCEN-S grant nubers N and N , and in part by Advanced Research and evelopent Activity under contract nuber MA94--C-984. Any opinions, findings and conclusions or recoendations expressed in this aterial are the author(s) and do not necessarily reflect those of the sponsor. 8. REFERENCES [] J. Ponte and W. B. Croft, A Language Modeling Approach to inforation retrieval. Proceedings of the 2 st annual international ACM SIGIR conference, , 998. [2] F. Song and W. B. Croft. A general language odel for inforation retrieva. Proceedings of the 22 nd annual international ACM SIGIR conference, , 999 [3]. Hiestra. Using language odels for inforation retrieval. Ph thesis, University of Twente, 2. [4] J. Lafferty and C. Zhai. ocuent language odels, query odels, and risk iniization for inforation retrieval. Proceedings of the 24 th annual international ACM SIGIR conference, -9, 2. [5] V. Lavrenko and W. B. Croft. evance-based language odels. Proceedings of the 24 th annual international ACM SIGIR conference, 2-27, 2. [6]. Miller, T. Leek, and R. Schwartz. A Hidden Markov Model inforation retrieval syste. Proceedings of the 22 nd annual international ACM SIGIR conference, 24-22, 999. [7] Swan, R. and Allan, J. Autoatic Generation of Overview Tielines. Proceedings of SIGIR 2 Conference, Athens, 49-56, 2. [8] Swan, R. and Jensen,. TieMines: Constructing Tielines with Statistical Models of Word Usage. Proceedings of K 2 Conference, 73-8, 2. [9] J. Allan, R. Gupta, and V. Khandelwal. Teporal Suaries of News Topics. Proceedings of ACM SIGIR conference, -8, 2.

7 [] J. Pustejovsky, TERQAS: Tie and Event Recognition for Question Answering Systes, ARA Workshop, MITRE, Boston (22). ( [] K. Wessel, W. Thijs, and H. joerd. The Iportance of Prior Probabilities for Entry Page Search, Proceddings of SIGIR 22, [2] [3] APPENIX: QUERIES USE IN EXPERIMENTS () Training set consists of following TREC queries: 32, 34, 36, 39, 32, 33, 333, 334, 34, 345, 35, 352, 355, 37, 378, 382, 385, 39, 395, 396 (2) Test set consists of following queries: 346, 4, 3, 356, 3, 337, 389, 37, 326, 329, 36, 376, 357, 387, 32, 347