Measuring the Ability of Score Distributions to Model Relevance

Transcription

1 Meauring the Ability of Score Ditribution to Model Relevance Ronan Cummin Department of Information Technology National Univerity of Ireland, Galway Abtract. Modelling the core ditribution of document returned from any information retrieval (IR) ytem i of both theoretical and practical importance. The goal of which i to be able to infer relevant and nonrelevant document baed on their core to ome degree of confidence. In thi paper, we how how the performance of mixture of core ditribution can be compared uing inference of query performance a a meaure of utility. We (1) outline method which can directly calculate average preciion from the parameter of a mixture ditribution. We (2) empirically evaluate a number of mixture for the tak of inferring query performance, and how that the log-normal mixture can model more relevance information compared to other poible mixture. Finally, (3) we perform an empirical analyi of the mixture uing the recall-fallout convexity hypothei. 1 Introduction Analying the document core returned from information retrieval (IR) ytem i a very ueful, yet challenging problem. Work in thi area can be dated back to the early day of IR [16]. Modelling the document core returned for different querie (and from different ytem) i an important tak becaue it ha been noticed that the ditribution of relevant document core i different than that of non-relevant document core. For example, if given the entire core ditribution (SD) returned from a ytem, the ditribution of relevant document could be accurately determined, it would be particularly ueful for automatic query performance prediction and/or meta-earch (fuion) tak [8, 4]. Regardle, the problem of correctly modelling the ditribution of relevant and non-relevant document remain an open, and theoretically important, area in IR. Over the lat decade, the predominant ditribution [1] for modelling relevant and non-relevant document core have been a normal and an exponential repectively. There ha been relatively little jutification a to why relevant and non-relevant document core hould be drawn from two different familie of ditribution. Neverthele, thee ditribution have bet fit the data for many year now. More recently, it ha been uggeted that the normal-exponential mixture i not theoretically valid under certain aumption [14], and in fact, a

2 more theoretically valid approach might be to model the core uing two gamma ditribution [9]. Thi paper deal with determining the bet ditribution for ue in a mixture model by uing the inference of performance (i.e. average preciion) a a meaure of utility, when relevance information i available. While the tak of inferring average preciion might be viewed a only one meaure of utility, it i one of the mot important tak in IR. Thi meaure of utility i very important a it i linked to a model ability to accurately model the relevance contained within, and i not only of practical concern but i of theoretical importance. We how that the log-normal ditribution i the bet ditribution to ue in a mixture model of core ditribution for both goodne-of-fit and utilty. The remainder of the paper i organied a follow: Section 2 review related work on modelling document core ditribution. Section 3 outline four mixture ditribution ued in thi paper, before the formulae for calculating the average preciion from a mixture model are introduced. Section 4 preent empirical reult comparing the four mixture model for a number of metric when relevance information i known. Section 5 preent an empirical analyi of the four mixture baed on Roberton recall-fallout convexity hypothei. Finally, ection 6 outline our concluion and future work. 2 Related Reearch In thi ection, we review related work in document core ditribution. 2.1 Related Work Early work into SD modeling ha hown that the ditribution of relevant document omewhat follow a normal curve [16]. Approache over the year have tried variou curve and fit to try and uncover the underlying ditribution. More recent work ha hown that modelling relevant and non-relevant document core uing a normal and exponential ditribution repectively, fit for the core at the head of the ranked lit (i.e. top 1000document) [1]. Indeed, thi ha been the predominant trend over recent year [14]. Other have addreed more theoretical apect of the underlying ditribution, and have developed hypothee under which certain ditribution can be theoretically rejected [14]. The aforementioned work develop a recall-fallout hypothei which tate that the recall-fallout curve for good ytem hould be upper convex and ha hown that if the probability ranking principle [13] hold, then certain ditribution hould be rejected on theoretical ground. Further work [2] ha hypotheied that a theoretically valid ditribution mut be able to approach the Dirac delta function (i.e. it mut be able to approach an impule under which the entire ma of document can reide). Some of the theoretical problem aociated with the infinite upport that ome ditribution allow were addreed recently [1] uing truncated form of ditribution. Some novel approache [10] to modelling the core ditribution

3 have ued multiple normal ditribution for the relevant document and a gamma ditribution of the non-relevant one. Thi approach ue thee ditribution becaue they are a good fit given the data. Important work in analying the generation proce(i.e. ranking function) of document core and their reultant ditribution ha alo been conducted [9]. On a practical note, reearch ha been conducted to ue the core ditribution for data fuion [12] and core threhold optimiation [1]. 2.2 Contribution Thi work ha a number of contribution. Firtly we how how average preciion can be inferred from a mixture ditribution. Secondly, we conduct an extenive evaluation of everal mixture model for a number of metric (one of which i the tak of inferring average preciion accurately), and advocate the ue of the log-normal model in particular. Interetingly, we how that the bet method of etimating parameter for the tak of inferring average preciion, i the method of moment(mme), rather than maximum likelihood etimate(mle). Finally, we how that the depite it uperior performance the log-normal mixture doe not adhere to Roberton recall-fallout convexity hypothei a well a the gamma mixture. 3 Model In thi ection, we preent four mixture ditribution ued in thi paper to model the core of both relevant and non-relevant document. 3.1 Aumption and Retriction Conider an IR ytem that retrieveareturned et of N document, and thu N core given a query (Q). Firtly, we aume that an IR ytem rank document independently of each other, in accordance with the probability ranking principle (PRP) [13]. While thi may not be true for certain ytem (e.g. for thoe that wih to promote diverity), it i a widely held principle in IR. Secondly, we aume a binary view of relevance. While core ditribution can be modelled a mixture of a multiple of differently graded relevance ditribution, thi work only model a binary view of relevance. We ued the following two criteria to elect the ditribution that are preented in ection 3.2. Firtly, under on the trong SD hypothei [2], the ditribution of both relevant and non-relevant document hould be able to approach Dirac delta function (thee ditribution are valid under that hypothei). And econdly, there i no theoretically valid reaon why relevant and non-relevant document hould be drawn from two different familie of ditribution, given that the document core of relevant and non-relevant document i generated uing the ame proce (ranking function) within an IR ytem.

4 3.2 Mixture Ditribution The ditribution that we conider are the normal ditribution (N), the exponential ditribution (E), the log-normal ditribution (L), and the gamma ditribution (G) [11]. For mot of the mixture outlined in thi ection both relevant and non-relevant document are modelled uing the ame ditribution. We only include the normal-exponential (N 1 E 0 ) mixture a it ha been ued in many tudie to model core ditribution for variou tak. Therefore, the next tep i to outline the mixture model that can be ued in conjunction with any ditribution. For mot mixture, we model both et of document uing the ame ditribution, where P( 1) i the pdf (probability denity function) for the core () of relevant document, and P( 0) i the pdf for the core of non-relevant document. Therefore, imilar to previou approache, the document core ditribution can be thought of a a mixture of relevant and non-relevant document a follow: P() = (λ) P( 1)+(1 λ) P( 0) (1) where λ = R N i the proportion of relevant document R in the entire returned et N. In practice, no form of document core normaliation i necearily needed for the upper limit for any of the ditribution. Although, negative value are not upported for the log-normal or gamma ditribution, for the information retrieval model ued in thi work, the iue of upporting negative core i not a problem in practice 1. In thi paper, we tudy four mixture. Table 1 outline the mixture and the parameter that need to be etimated for each model. For the parameter of each model, we ue the ubcript 1 to imply that the parameter i ued with the ditribution of relevant document core, wherea we ue the ubcript 0 to imply that the parameter i ued with the ditribution of non-relevant document core. For three of the mixture, there are a total of five parameter (i.e. the mixture parameter, two parameter to model the relevant core and two parameter to model the non-relevant core), while the normal-exponential model (N 1 E 0 ) ha only four parameter. Thi i important for comparion purpoe, a model (and ditribution) with more parameter have more flexibility in modelling the oberved data. Therefore, ome model may have le flexible in term of their ability to model core from different ytem. Although we have included the normal-exponential (N 1 E 0 ) model in thi tudy, it i in the author opinion that document core of relevant and non-relevant document hould not be drawn from two different familie ofditribution. For the N 1 E 0 and N 1 N 0 mixture, the MME(method of moment etimate) and MLE(maximum likelihood) etimate are equivalent. However, for the L 1 L 0 and G 1 G 0 mixture, the MME and MLE etimate will lead to different parameter etting. 1 Theoccurrence ofnegativecore caneaily beovercome inpractice byimplyhifting all core by ome contant factor. In theory, a core are generated from termfrequency evidence (bounded by zero), there are ome argument a to why negative core hould not occur in an IR model.

5 Label Table 1. Compoition of Mixture Relevant Non-Relevant # of parameter parameter MME = MLE N 1E 0 Normal Exponential 4 µ 1,σ 1,β 0,λ ye N 1N 0 Normal Normal 5 µ 1,σ 1,µ 0,σ 0,λ ye L 1L 0 Log-Normal Log-Normal 5 µ 1,σ 1,µ 0,σ 0,λ no G 1G 0 Gamma Gamma 5 k 1,θ 1,k 0,θ 0,λ no 3.3 Inferring Average Preciion In thi ection, we will how how average preciion (a tandard metric for the effectivene of a query) can be calculated directly from the mixture of continuou ditribution. Firtly, it i worth noting that average preciion i an informative meaure. A average preciion can be viewed geometrically a the area under the preciion-recall curve [3] 2, we know that it ummarie the performance over a large portion of the ranked lit, and therefore, convey a broad view of the effectivene of a query. Secondly, it i a table meaure [5], and i probably the mot prevalent metric of both query and ytem performance ued in IR literature. The intereted reader i referred to reearch which trongly outline the theoretical importance of average preciion [15]. A recall i the proportion of relevant returned document compared to the entire number of relevant document, the recall at core can be defined a follow: recall() = λ P( 1) d λ = P( 1) d (2) which i the cumulative denity function (cdf) of the ditribution of relevant document (viewed from ). Under the ditribution outlined earlier for our model, we know that recall() will vary between 0 and 1, (i.e. when = 0, recall() = 1 a enured by the cdf). Similarly, the preciion at (the proportion of relevant returned document over the number of returned document) can be defined a follow: λ P( 1) preciion() = (λ) P( 1)+(1 λ) P( 0) (3) Now that we can calculate the preciion and recall at any core in the range [0 : ], we can create a preciion-recall curve. Furthermore, a average preciion can be etimated geometrically by the area under the preciion-recall curve [3], the average preciion (avg.prec) of a query can be calculated a follow: avg.prec() = 1 0 preciion() dr() (4) 2 Preliminary experiment have hown that the linear correlation between the actual average preciion and the area under the interpolated preciion-recall curve i greater than 0.95

6 where r() = recall() which i in the range [0:1]. Thi formulation i an elegant and intuitive way of calculating average preciion uing the core ditribution. A thee expreion are not cloed-form, they can be calculated uing relatively imple geometric numerical integration method. 4 Mixture Performance In thi ection we perform a comparative analyi of the four mixture model acro a number of different IR model (i.e. vector pace, claic probabilitic, language model, learned model, and axiomatic model). Firt, however, we will motivate our choice of comparion metric. 4.1 Goodne-of-Fit, Correlation, and RMSE Uually, the performance of a mixture model i determined by how well the model fit actual data. For different field of tudy and for different problem, different metric may be applicable. Uually, goodne-of-fit tet (e.g. Kolmogorov- Smirnov tet) are ued to either accept or reject certain model a a good fit. However, in IR, it i well-known that document, and therefore document core, at the head of a ranked lit are more important than thoe further down the lit 3. Thee goodne-of-fit tet do not make a ditinction between obervation (i.e. core) at variou location and they do not meaure the amount of relevant information that can be correctly maintained in the model. We propoe that better mixture model are better able to model the information regarding relevance. An intuitive way of meauring thi i by trying to infer the average preciion of a query uing the model (and it known parameter). Average preciion i a natural candidate for capturing the performance (a dicued earlier). Therefore, over a et of topic, the correlation between the inferred average preciion from the mixture model and the actual average preciion of the query from the IR ytem, give u a meaure of how much relevance information i contained in each model. From an information theoretic point of view, it alo give u an indication of how much relevance information i lot when modelling each ranking with a particular mixture model. Table 2. Tet Collection Detail Collection # doc # topic range AP 242, Tet FT 210, WT2G 221, WT10G 1,692, Looking at only a part of the ranked lit (e.g. document up to rank 1000) doe not effectively olve thi problem.

7 4.2 Comparative Analyi We now compare the four mixture model introduced earlier (i.e. Table 1) over a range of IR ytem and etting. Different ditribution may better be able to model different IR ytem and o for a broader comparion, we compared the four mixture acro five IR model. We choe the vector pace model uing pivoted document normaliation (PIV) [7], the probablitic model (BM25) [7], a language modelling (LM) approach (Jelinek-Mercer moothing) [17], a learned approach (ES) [6], and the axiomatic approach (F2EXP) [7], a thee repreent a broad range of claical and more modern ranking function. Table 2 how the tet collection ued in thi reearch. Goodne-of-fit Table 3 how the Kolmgorov-Smifnoff D-tatitic 4 (a meaure of goodne-of-fit) on each of the collection averaged over the five ytem. The D-tatitic meaure the maximum ditance between the cumulative denity function of the theoretical ditribution (i.e. one of the mixture) and the empirical ditribution (i.e. the actual core). Firtly, we can ee that Table 3 how that the log-normal model ha a ignificantly 5 better fit compared to the gamma model on two collection for the entire returned et of document core. The reult alo how that the log-normal model fit non-web collection very well, but the gamma model ha a better fit for ome IR ytem on web collection. The normal-exponential model i the third bet model in term of fit, while the normal-normal model i particularly poor. We can alo ee from Table 3 that the MLE parameter etimation technique provide better fit, in general, than MME. Table 3. Average Kolmgorov-Smifnoff D-tatitic for querie acro all ytem for title querie uing entire returned et of document core MME MLE Collection N 1E 0 N 1N 0 L 1L 0 G 1G 0 L 1L 0 G 1G 0 AP FT WT2G WT10G A the parameter of the model are etimated from the oberved ample, the critical value of the Kolmgorov-Smifnoff tet are invalid. However, we ue the D-tatitic a a relative meaure to compare the mixture, and not a a tatitical tet to accept or reject the validity of the ditribution. 5 x denote that the tatitic i ignificantly lower than the next bet model uing the ame parameter etimation technique for x of the five ytem.

8 Table 4. Average Spearman (and Pearon in parenthee) correlation between mixture model inferred average preciion and actual average preciion acro five IR ytem for title querie uing entire returned et of document MME MLE Mixture N 1E 0 N 1N 0 L 1L 0 G 1G 0 L 1L 0 G 1G 0 AP 0.47 (0.26) 0.56 (0.32) 0.89 (0.84) 0.84 (0.76) 0.80 (0.71) 0.77 (0.66) FT 0.33 (0.24) 0.55 (0.30) 0.89 (0.81) 0.86 (0.75) 0.83 (0.75) 0.80 (0.67) WT2G 0.45 (0.32) 0.49 (0.35) 0.83 (0.83) 0.81 (0.81) 0.72 (0.67) 0.73 (0.70) WT10G 0.39 (0.33) 0.40 (0.07) 0.74 (0.61) 0.66 (0.55) 0.62 (0.46) 0.58 (0.44) Correlation and RMSE Now we analye the amount of relevance information that can be correctly contained within each mixture model acro the five IR ytem uing correlation meaure. Uing the MME and MLE approache we can etimate the five parameter in each mixture model auming relevance information i known (i.e. labelled data). We then compare the correlation of the inferred average preciion (calculated from equation 4) for the mixture model with the actual average preciion from the IR ytem in quetion. Table 4 how the average Spearman and Pearon correlation of the four mixture model averagedacro the five ytem 6. Firtly, it i worth noting that the correlation coefficient for ome of the mixture are quite high, indicating that much of the information regarding average preciion (relevance) are correctly modeled by ome of the mixture. We can alo ee that the mixture model compried of a normal and exponential (i.e. the predominant model over the lat decade) i the lowet performing model of the four that we have tudied. The normal-normal model outperform the exponential-normal model in term of utility depite having a wore fit (ee Table 3). In general, we can alo ee that the log-normal mixture model tend to outperform the gamma model acro a variety of etting and parameter etimation technique (i.e. for both MME and MLE etimate). In general, the reult how that the log-normal model i the more general and conitent model for preerving relevance information acro a variety of IR ytem. Table 5 how the root mean quared error (RMSE) 7 of the inferred average preciioncomparedtotheactualaveragepreciionforaetofquerieforboththe BM25 and LM ytem (the other ytem teted howed comparable reult). We can ee that the actual average preciion predicted by the log-normal model i cloer to the true average preciion. While the RMSE i not of major importance in term of the predictive quality of a model, it doe inform u that the raw output of the log-normal mixture model i cloer to the actual average preciion 6 The bold font indicate that the average correlation i higher than the next highet acro all five ytem. Statitical tet do not how any difference between the top two performing mixture model. Statitical tet do how a higher correlation for the gamma and log-normal model compared to the other mixture. 7 The denote that the reduced error i ignificant compared to the gamma mixture. A Wilcoxon ranked ign tet at the 0.01 level wa ued.

9 of a query. The RMSE reult of all other IR ytem are comparatively imilar to thoe in Table 5. One reaon for thi error i that the formulae given for inferring average preciion from core ditribution (Section 3.3) will actually over-etimate the actual average preciion value on TREC data due to the fact that recall i calculated a the number of relevant document in the returned et, rather than the total number of relevant document in the collection. Table 5. RMSE of Inferred Average Preciion (uing MME) compared to two Sytem (BM25 and LM) Actual Average Precion for title querie uing entire returned et of document core Mixture N 1E 0 N 1N 0 L 1L 0 G 1G 0 N 1E 0 N 1N 0 L 1L 0 G 1G 0 BM25 LM AP FT WT2G WT10G MME v MLE Another interetingpoint i that the MME approachto parameter etimation conitently outperform the MLE approach in term of utility (i.e. for the tak of inferring performance a meaured by the correlation in Table 4). However, when all ample obervation are treated equally (a for goodne-of-fit tet), the D-tatitic in Table 3 how that model derived from MLE are cloer to the oberved ample. Thi provide further proof that the correlation coefficient and goodne of fit tet meaure different apect. A we are dealing with IR ytem, and model of relevance, we argue that a tandard meaure of utility i more apt. 5 Recall-Fallout Convexity Analyi Of the mixture tudied in thi paper, we have empirically determined that the mixture of two log-normal i one of the better mixture for modelling document core for a number evaluation metric. Furthermore, our reult ugget that it i very robut and can accurately model ranking returned from many ytem. However, it i unclear if thi mixture adhere to ueful theoretical propertie. In thi ection, we analye all of the mixture uing the recall-fallout convexity hypothei [14]. Interetingly, we how that the gamma mixture violate the recall-fallout hypothei le often than the log-normal mixture near the head of the ranked lit (i.e. where it i more important). 5.1 Locating Point of Non-Convexity The recall-fallout hypothei tate that a we travere a ranked-lit, the recall hould alway be greater than fallout. Thi eem theoretically jutifiable, a

10 IR ytem hould at leat provide a better than random ranking. Therefore, when modelling document ranking a continuou ditribution, the recall-fallout hypothei can be more formally tated a P( 1) d > P( 0) d for all. A detailed analyi of the recall-fallout convexity hypothei for all of the mixture tudied in thi paper (except the two log-normal mixture) can be found in the original work [14]. Uing notation imilar to the original work, the convexity condition that mut be atified to enure that recall i greater than fallout for all core, can be written a follow: where g 1 () > g 0 () (5) g() = 1 df() f() d where f() i the probability denity function of a particular ditribution. Now auming thi hypothei to be valid, it would be intereting to ee how cloely the better mixture model adhere it. (6) Gamma Mixture Therefore, a g() = ((k 1)/) 1/θ for the gamma mixture [14], the core at which the condition i violated i found by implifying the following: which implifie to k θ 1 = k θ 0 (7) = θ 1θ 0 k 0 θ 1 θ 0 k 1 θ 1 θ 0 (8) We can ee that if θ 1 = θ 0, there are no root for, and o no violation occur. Furthermore, if k 1 = k 0, = 0 and o, the violation occur at the point at which both recall and fallout are 1 (which i acceptable). For the two-gamma mixture, if > 0, the violation occur at a core that can be encountered by the mixture, otherwie the violation doe not occur. Log-NormalMixture Forthelog-normalmixtureg() = (µ log() σ 2 )/( σ 2 ), and therefore, the core at which the convexity condition i violated i found at: = e (µ1σ2 0 µ0σ2 1 )/(σ2 0 σ2 1 ) (9) by following a imilar implification proce a the gamma mixture. We can ee that if σ 1 = σ 0, the function ha no root, and therefore, no violation (imilar to the normal ditribution [14]). If the variance are not exactly equal, a violation of the convexity condition, will occur at a core above zero. The core at which a violation occur can be tranlated to a point of recall uing equation (2).

11 5.2 Empirical Reult and Dicuion We analyed the four mixture model by calculating the point of recall at which the convexity condition wa violated for each query on the tet collection. It i reaonable to aume that a violation at the head (i.e. low point of recall) of the ranked lit i more eriou than if it occur at high recall. However, if the convexity condition i violated at a core that i rarely, or never, encountered by an IR metric (at high recall), it i deemed le eriou. Table 6 report the average point of recall at which a violation of the convexity condition occur for a et of querie averaged acro the five IR ytem. The reult in Table 6 are from the four model when uing MME a the parameter etimation technique. In general, we can ee that the two-gamma model i the more theoretically ound a violation occur, on average, at a higher point of recall (e.g. at a lower core ). Surpriingly, violation occur at a low point of recall for the log-normal model (even lower than the two-normal model), which ugget that it i theoretically le ound that either the two-gamma model or the two-normal model. The exponential-normal mixture ha violation at both end of the relevant ditribution (i.e. both high and low recall) 100% of the time, and therefore, we can ee from Table 6 that the violation occur very early on in the ranking (i.e. low point of recall). The reult from Table 6 confirm previou analyi [14] with regard to many of thee model. The averagereult acro the five IR ytem in Table 6 are highly repreentative of each individual ytem. More work i needed to undertand the reaon for the apparent hortcoming in the theoretical behaviour of the two log-normal model (epecially a it outperform other mixture in term of goodne-of-fit and utility). Table 6. Average recall at which convexity violation occur for different model Mixture N 1E 0 N 1N 0 L 1L 0 G 1G 0 AP FT WT2G WT10G Concluion In thi work, we have performed a comparative analyi of different ditribution that comprie mixture for document core ditribution in IR ytem. We have determined that the log-normal ditribution i the bet performing model in term of both it accuracy in inferring average preciion, and it goodne-of-fit. The log-normal model ha been ued in relatively few practical work. Interetingly, we have hown depite it good performance the log-normal model i theoretically le ound than the two-gamma model toward the head of a ranking.

12 Intereting future work would be to create mixture model that unconditionally adhere to the recall-fallout convexity hypothei (e.g. by enuring σ 1 = σ 0 for the two log-normal model) and then compare the utility of thoe valid model. Reference 1. Avi Arampatzi, Jaap Kamp, and Stephen Roberton. Where to top reading a ranked lit?: threhold optimization uing truncated core ditribution. In SIGIR, page , Avi Arampatzi and Stephen Roberton. Modeling core ditribution in information retrieval. Inf. Retr., 14(1):26 46, Javed A. Alam and Emine Yilmaz. A geometric interpretation and analyi of r-preciion. In CIKM, page , Chritoph Baumgarten. A probabilitic olution to the election and fuion problem in ditributed information retrieval. In ACM SIGIR conference on Reearch and development in information retrieval, SIGIR 99, page , New York, NY, USA, ACM. 5. Chri Buckley and Ellen M. Voorhee. Evaluating evaluation meaure tability. In SIGIR, page 33 40, Ronan Cummin and Colm O Riordan. Learning in a pairwie term-term proximity framework for information retrieval. In SIGIR, page , Hui Fang and ChengXiang Zhai. An exploration of axiomatic approache to information retrieval. In SIGIR, page , Ben He and Iadh Ouni. Query performance prediction. Inf. Syt., 31(7): , Evangelo Kanoula, Kehi Dai, Virgiliu Pavlu, and Javed A. Alam. Score ditribution model: aumption, intuition, and robutne to core manipulation. In SIGIR, page , Evangelo Kanoula, Virgiliu Pavlu, Kehi Dai, and Javed A. Alam. Modeling the core ditribution of relevant and non-relevant document. In ICTIR, page , N Hating M Evan and B Peacock. Statitical ditribution, third edition. Meaurement Science and Technology, 12(1):117, R. Manmatha, T. Rath, and F. Feng. Modeling core ditribution for combining the output of earch engine. In Proceeding of the 24th annual international ACM SIGIR conference on Reearch and development in information retrieval, SIGIR 01, page , New York, NY, USA, ACM. 13. C. J. Van Rijbergen. Information Retrieval. Butterworth-Heinemann, Newton, MA, USA, 2nd edition, Stephen Roberton. On core ditribution and relevance. In Proceeding of the 29th European conference on IR reearch, ECIR 07, page 40 51, Berlin, Heidelberg, Springer-Verlag. 15. Stephen E. Roberton, Evangelo Kanoula, and Emine Yilmaz. Extending average preciion to graded relevance judgment. In Proceeding of the 33rd international ACM SIGIR conference on Reearch and development in information retrieval, SIGIR 10, page , New York, NY, USA, ACM. 16. John A. Swet. Information retrieval ytem. Science, 141(3577): , Chengxiang Zhai and John Lafferty. A tudy of moothing method for language model applied to information retrieval. ACM Tran. Inf. Syt., 22: , April 2004.